Skill: aduo6668/amazon-review-workbook (version 1.0.3)

--- name: amazon-review-workbook description: Collect all customer reviews from an Amazon product URL or product-reviews URL through a logged-in Chrome session on port 9222, export a 14-column factual workbook, optionally fill translations through DeepLX, and then help the model tag the rows into a final delivery-ready spreadsheet. Use when the user sends an Amazon link and wants review scraping, competitor review analysis, review export, or a delivery-ready spreadsheet with usernames, review links, review time, helpful votes, translation, summary, sentiment, categories, and tags. --- # Amazon Review Workbook Turn an Amazon product or review link into a two-phase delivery workbook. This skill is designed to be portable: the scripts live inside the skill folder and do not depend on `dashcamauto` or any other local repo. ## Quick Path 1. If this is the first run on a machine, read [references/setup.md](references/setup.md). 2. Run a quick health check: ```bash python scripts/amazon_review_workbook.py doctor --url "<amazon-url>" ``` 3. Run factual collection: ```bash python scripts/amazon_review_workbook.py intake --url "<amazon-url>" --output-dir "<workspace>/amazon-review-output" ``` 4. If DeepLX is configured and reachable, fill `评论中文版`: ```bash python scripts/amazon_review_workbook.py translate --input-json "<workspace>/amazon-review-output/amazon_<asin>_review_rows_factual.json" --output-dir "<workspace>/amazon-review-output" ``` 5. Check coverage before deciding whether keyword expansion is worth the extra requests: ```bash python scripts/amazon_review_workbook.py coverage-check --url "<amazon-url>" --db-path "<workspace>/amazon-review-output/amazon_review_cache.sqlite3" ``` 6. Build canonical tags and a lightweight tagging payload: ```bash python scripts/amazon_review_workbook.py taxonomy-bootstrap --input-json "<workspace>/amazon-review-output/amazon_<asin>_review_rows_translated.json" --output-dir "<workspace>/amazon-review-output" python scripts/amazon_review_workbook.py prepare-tagging --input-json "<workspace>/amazon-review-output/amazon_<asin>_review_rows_translated.json" --output-dir "<workspace>/amazon-review-output" --canonical-tags-json "<workspace>/amazon-review-output/canonical_tags.json" ``` `taxonomy-bootstrap` is only for building a stable canonical vocabulary for the batch. `prepare-tagging` consumes the full factual or translated JSON and emits a trimmed `*_tagging_input.json` that contains pending rows only plus cache metadata. Do not use that trimmed file as the merge source. 7. Read [references/tagging-guidelines.md](references/tagging-guidelines.md), let the model fill only the pending rows in a separate labels JSON, then merge the labels back into the full base JSON and build the final workbook: ```bash python scripts/amazon_review_workbook.py merge-build --base-json "<workspace>/amazon-review-output/amazon_<asin>_review_rows_translated.json" --labels-json "<workspace>/amazon-review-output/amazon_<asin>_labels.json" --output-dir "<workspace>/amazon-review-output" --taxonomy-version "v1" --strict ``` ## Workflow ### 1. Verify prerequisites - Confirm `doctor` reports a valid `asin`. - Confirm `chrome_debug_ready` is `true`. - If you plan to use `translate`, confirm `deeplx_env_ready` is `true`. - If `deeplx_reachable` is `false`, do not block the workflow; let the model fill `评论中文版` during tagging. If any of these fail, read [references/setup.md](references/setup.md) before continuing. ### 2. Use the smallest command that fits - For raw review collection only: use `collect` - For factual extraction plus workbook scaffolding: use `intake` - For deciding whether a keyword pass is still needed: use `coverage-check` - For rebuilding the tuned keyword state from historical data: use `keyword-autotune` - For machine translation of `评论中文版`: use `translate` - For canonical tag sampling: use `taxonomy-bootstrap` - For cache-aware lightweight model input: use `prepare-tagging` - For writing the final labeled workbook: use `merge-build` Examples: ```bash python scripts/amazon_review_workbook.py collect --url "<amazon-url>" --output-dir "<workspace>/amazon-review-output" python scripts/amazon_review_workbook.py translate --input-json "<workspace>/amazon-review-output/amazon_<asin>_review_rows_factual.json" --output-dir "<workspace>/amazon-review-output" python scripts/amazon_review_workbook.py coverage-check --url "<amazon-url>" --db-path "<workspace>/amazon-review-output/amazon_review_cache.sqlite3" python scripts/amazon_review_workbook.py keyword-autotune --output-dir "<workspace>/amazon-review-output" --db-path "<workspace>/amazon-review-output/amazon_review_cache.sqlite3" python scripts/amazon_review_workbook.py taxonomy-bootstrap --input-json "<workspace>/amazon-review-output/amazon_<asin>_review_rows_translated.json" --output-dir "<workspace>/amazon-review-output" python scripts/amazon_review_workbook.py prepare-tagging --input-json "<workspace>/amazon-review-output/amazon_<asin>_review_rows_translated.json" --output-dir "<workspace>/amazon-review-output" --canonical-tags-json "<workspace>/amazon-review-output/canonical_tags.json" python scripts/amazon_review_workbook.py merge-build --base-json "<workspace>/amazon-review-output/amazon_<asin>_review_rows_translated.json" --labels-json "<workspace>/amazon-review-output/amazon_<asin>_labels.json" --output-dir "<workspace>/amazon-review-output" --taxonomy-version "v1" --strict ``` ### 3. Keep the workbook stable The factual and final workbooks always use the 14-column schema in [references/output-schema.md](references/output-schema.md). Do not silently add or remove columns. If a field is unavailable from the page, leave it blank rather than inventing a value. ### 4. Tag rows only after grounding on the factual file The model should not invent from the product page alone. Ground semantic tagging on the factual JSON/workbook created by `intake` or `translate`. Keep the two JSON shapes distinct: - `*_tagging_input.json` from `prepare-tagging` is the cropped machine prompt payload for the model - `--base-json` for `merge-build` must be the full factual/translated record set, not the cropped tagging payload - `--labels-json` is the model's completed semantic output for the pending rows only If `translate` prints `translation_mode=model_fallback`, fill `评论中文版` in the same tagging pass instead of waiting for DeepLX. Use [references/tagging-guidelines.md](references/tagging-guidelines.md) when filling: - `评论概括` - `情感倾向` - `类别分类` - `标签` - `重点标记` The preferred fast path is: 1. `taxonomy-bootstrap` to build a canonical tag vocabulary for this batch 2. `prepare-tagging` to create a minimal pending-row payload 3. model labeling only for pending rows, written into a separate labels JSON 4. `merge-build` to update cache and export the final workbook from the full base JSON ## Collection Defaults - `intake` and `collect` no longer run keyword expansion implicitly in `deep` mode. `deep` now means the 18 combo pass only. - Run `coverage-check` after intake to compare current rows vs Amazon's visible `reviews` count before deciding to spend more requests. - Use `--keywords` only when you explicitly want a keyword pass. - Use `--keywords` with no values to run the built-in keyword preset for the selected `--keyword-profile`. - Use `--keywords foo bar baz` to provide an explicit keyword list. - Default pacing now inserts a `2.5s` gap between combos/keywords to reduce rate-limit risk. - Built-in profiles: - `generic`: universal consumer-product terms - `electronics`: universal terms + common app/setup/hardware terms - `dashcam`: electronics profile + recording/night/parking/GPS/Wi-Fi/mount terms - Default keyword reuse policy is `successful`: keywords that have produced results before are skipped on later runs; recent zero-result keywords are also suppressed for `72h` to avoid immediate retries. - If you really want to brute-force rerun every keyword, use `--keyword-reuse-scope none`. - A tuned state file at `<output-dir>/keyword_tuning_state.json` is now read automatically when present, and refreshed after keyword runs so the skill gradually reorders towards higher-yield terms. - `keyword-autotune` can also ingest old keyword-run JSON reports via `--report-glob` to seed the tuned state from historical experiments. ## Failure Boundaries Do not claim success if any of these is true: - The script did not reach a real review page. - The expected XLSX/CSV for the current phase was not generated. - Review links, review time, or helpful votes were guessed rather than extracted. - The model tagged rows without first grounding on the factual JSON/workbook. - The cropped `*_tagging_input.json` was used as `--base-json` for `merge-build`. - The model re-labeled rows that were already cached for the same taxonomy version. - The workflow still claims a 13-column contract after `评论用户名` was added as a real output column. ## Resources - [references/setup.md](references/setup.md): first-run machine setup and environment requirements - [references/output-schema.md](references/output-schema.md): fixed 14-column workbook contract - [references/tagging-guidelines.md](references/tagging-guidelines.md): semantic labeling rules after factual collection - [scripts/amazon_review_workbook.py](scripts/amazon_review_workbook.py): portable CLI for doctor/collect/intake/coverage-check/keyword-autotune/translate/taxonomy-bootstrap/prepare-tagging/merge-build - [scripts/review_delivery_schema.py](scripts/review_delivery_schema.py): workbook schema, normalization, and XLSX/CSV writer - [scripts/deeplx_translate.py](scripts/deeplx_translate.py): optional DeepLX translation helper - [scripts/label_workflow.py](scripts/label_workflow.py): cache, heuristics, bootstrap, and merge logic for faster labeling

path: agents/openai.yaml
content: interface:
  display_name: "Amazon Review Workbook"
  short_description: "Grab Amazon reviews and export a delivery-ready workbook."
  default_prompt: "Use $amazon-review-workbook to collect reviews from this Amazon link, fill the factual workbook, run coverage-check before any keyword expansion, and then build the final labeled 14-column workbook."


path: LICENSE.zh-CN.md
content: # MIT 许可证（中文对照版）

以下内容是 MIT License 的中文对照说明，便于阅读理解。  
如中文表述与英文原文存在差异，请以仓库根目录中的 [LICENSE](./LICENSE) 英文原文为准。

版权所有 (c) 2026 aduo6668

特此免费授予任何获得本软件及相关文档文件（以下简称“软件”）副本的人不受限制地处置本软件的权利，包括但不限于：

- 使用
- 复制
- 修改
- 合并
- 发布
- 分发
- 再许可
- 销售软件副本

以及允许被提供本软件的人这样做，但须满足以下条件：

软件的所有副本或重要部分都必须包含上述版权声明和本许可声明。

本软件按“原样”提供，不提供任何形式的明示或默示担保，包括但不限于适销性、特定用途适用性和不侵权的担保。在任何情况下，作者或版权持有人均不对因本软件或本软件的使用或其他交易所引起、产生或与之相关的任何索赔、损害或其他责任负责，无论是在合同诉讼、侵权诉讼还是其他诉讼中。


path: README.md
content: # Amazon Review Workbook Skill

这是一个可移植的 Amazon 评论抓取与表格交付 skill。

它可以完成这些事情：

- 通过已登录的 Chrome 会话抓取 Amazon 商品评论
- 导出固定 14 列的事实工作簿
- 可选接入 DeepLX 自动补全 `评论中文版`
- 生成轻量标注输入，交给模型补全语义字段
- 将模型标注结果合并回完整工作簿
- 在追加抓取前先做覆盖率检查
- 根据历史跑数自动调优关键词词表

## 仓库结构

- `README.md`：仓库使用说明
- `SKILL.md`：给 agent 环境使用的 skill 说明
- `references/`：配置、输出契约、标注规则
- `scripts/`：核心 CLI 和处理脚本
- `agents/openai.yaml`：默认 agent prompt 元数据
- `tests/`：轻量回归测试

## 环境要求

- Python 3.11 及以上
- 一个已经登录 Amazon 的 Chrome，会话开启远程调试端口 `9222`
- 需要的 Python 包：

```bash
pip install pandas openpyxl requests websocket-client
```

- 如果想先自动翻译，再做语义标注，可选配置 DeepLX

## 配置方法

### 1. 启动带远程调试的 Chrome

Windows：

```powershell
"$env:ProgramFiles\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222 --user-data-dir="$env:LOCALAPPDATA\Google\Chrome\User Data"
```

启动后确认这个 Chrome 会话里已经登录了 Amazon。

### 2. 可选配置 DeepLX

把 `.env.example` 复制为本地 `.env`，或者直接设置环境变量：

```env
DEEPLX_API_URL=https://your-deeplx-host/translate
DEEPLX_API_KEY=your-optional-key
```

## 快速开始

### 健康检查

```bash
python scripts/amazon_review_workbook.py doctor --url "<amazon-url>"
```

### 抓取事实工作簿

```bash
python scripts/amazon_review_workbook.py intake --url "<amazon-url>" --output-dir "./amazon-review-output"
```

### 在继续追加抓取前先看覆盖率

```bash
python scripts/amazon_review_workbook.py coverage-check --url "<amazon-url>" --db-path "./amazon-review-output/amazon_review_cache.sqlite3"
```

### 可选：先补 `评论中文版`

```bash
python scripts/amazon_review_workbook.py translate --input-json "./amazon-review-output/amazon_<asin>_review_rows_factual.json" --output-dir "./amazon-review-output"
```

### 生成标注输入

```bash
python scripts/amazon_review_workbook.py taxonomy-bootstrap --input-json "./amazon-review-output/amazon_<asin>_review_rows_translated.json" --output-dir "./amazon-review-output"
python scripts/amazon_review_workbook.py prepare-tagging --input-json "./amazon-review-output/amazon_<asin>_review_rows_translated.json" --output-dir "./amazon-review-output"
```

### 合并标注结果，导出最终工作簿

```bash
python scripts/amazon_review_workbook.py merge-build --base-json "./amazon-review-output/amazon_<asin>_review_rows_translated.json" --labels-json "./amazon-review-output/amazon_<asin>_labels.json" --output-dir "./amazon-review-output" --taxonomy-version "v1" --strict
```

## 关键词策略

现在的关键词搜索已经做了收束和提速：

- `deep` 模式默认只跑 combo，不自动跑关键词
- 只有显式传 `--keywords` 才会进入关键词阶段
- `--keywords` 不带值时，会使用内置 profile
- 内置 profile：
  - `generic`
  - `electronics`
  - `dashcam`
- 默认复用策略是 `successful`
  - 历史上搜出过新增评论的关键词会被跳过
  - 近期 0 命中的关键词会临时抑制，避免短时间重复撞墙

示例：

```bash
python scripts/amazon_review_workbook.py intake --url "<amazon-url>" --output-dir "./amazon-review-output" --keywords --keyword-profile electronics
```

```bash
python scripts/amazon_review_workbook.py intake --url "<amazon-url>" --output-dir "./amazon-review-output" --keywords --keyword-profile dashcam --keyword-reuse-scope none
```

## 关键词自动调优

可以根据历史 SQLite 缓存和旧关键词实验报告，生成或刷新调优状态：

```bash
python scripts/amazon_review_workbook.py keyword-autotune --output-dir "./amazon-review-output" --db-path "./amazon-review-output/amazon_review_cache.sqlite3" --report-glob "./reports/*keywords*.json"
```

它会生成 `keyword_tuning_state.json`，之后关键词阶段会自动优先读取这个调优结果。

## 主要命令

- `doctor`
- `collect`
- `intake`
- `coverage-check`
- `keyword-autotune`
- `translate`
- `taxonomy-bootstrap`
- `prepare-tagging`
- `merge-build`
- `summary`

## 发布说明

这个仓库的定位是“可公开复用的 skill 源码仓库”，不是私有工作目录镜像。

### 发布边界

发布前请确认以下内容不会进入仓库：

- `.env`
- `amazon-review-output/`
- SQLite 缓存文件
- `label_cache.jsonl`
- 真实评论导出表格
- 一次性本地调试脚本或临时分析产物

### 推荐发布流程

1. 先跑测试：

```bash
python -m unittest discover -s tests -p "test_*.py"
```

2. 确认 `.gitignore` 已覆盖本地产物。

3. 检查 staged 文件，只保留这些类型：

- `README.md`
- `SKILL.md`
- `references/`
- `scripts/`
- `tests/`
- `agents/openai.yaml`
- `.env.example`
- `LICENSE`

4. 提交前重点检查：

- 是否有真实评论数据
- 是否有本机绝对路径
- 是否有 API Key / token / cookie
- 是否把 `amazon-review-output` 里的实际样本带进去了

5. 提交并推送：

```bash
git add .
git commit -m "your message"
git push origin main
```

### 版本建议

如果改动较大，建议按下面的粒度发版：

- `patch`：修 bug、改文档、小幅参数调整
- `minor`：新增命令、新增 profile、新增非破坏性能力
- `major`：改输出契约、改工作流主路径、移除旧参数

### 发布后建议检查

- 仓库首页 README 是否可直接教会新人配置
- `LICENSE` 是否存在
- `git clone` 后是否能按 README 跑通 `doctor --help`
- `SKILL.md`、CLI 参数、README 三者是否一致

## 输出契约

事实工作簿和最终工作簿都使用固定 14 列：

1. `序号`
2. `评论用户名`
3. `国家`
4. `星级评分`
5. `评论原文`
6. `评论中文版`
7. `评论概括`
8. `情感倾向`
9. `类别分类`
10. `标签`
11. `重点标记`
12. `评论链接网址`
13. `评论时间`
14. `评论点赞数`

详细字段规则见 [references/output-schema.md](references/output-schema.md)。

## 测试

```bash
python -m unittest discover -s tests -p "test_*.py"
```

## 说明

- `.env`、SQLite 缓存、真实评论导出文件、本地输出产物默认都不会提交
- 这个仓库发布的是可复用 skill，不包含私有评论数据


path: references/output-schema.md
content: # Output Schema

This skill uses two different JSON shapes and one fixed 14-column workbook:

## 1. Machine / factual JSON

This is the full review row data produced by `collect`, `intake`, or `translate`.

- It can contain factual fields only
- It can also carry helper fields such as `machine` or `delivery` wrappers
- It is the source that `merge-build` should read from

## 2. Labels JSON

This is the model's completed semantic output for the pending rows only.

- It should contain the final semantic fields, not the cropped `prepare-tagging` payload
- It may be a JSON array or an object with `items`
- It is merged back into the full base JSON by `merge-build`

## 3. Delivery workbook

The final workbook always exports these 14 columns, in order:

1. `序号`
2. `评论用户名`
3. `国家`
4. `星级评分`
5. `评论原文`
6. `评论中文版`
7. `评论概括`
8. `情感倾向`
9. `类别分类`
10. `标签`
11. `重点标记`
12. `评论链接网址`
13. `评论时间`
14. `评论点赞数`

## Field Rules

- `评论用户名`: Fill only from the visible review card; if unavailable, leave blank.
- `国家`: Fill only when the page explicitly reveals country/region.
- `星级评分`: Fill only when the page explicitly reveals a star rating.
- `评论原文`: Keep the original review language. If the body is empty, title fallback is allowed.
- `评论中文版`: Faithful Chinese translation.
- `评论概括`: One-sentence summary.
- `情感倾向`: Must be `Positive`, `Negative`, or `Neutral`.
- `类别分类`: Use 1-2 high-signal categories.
- `标签`: Short Chinese tags, ideally <= 10 chars each.
- `重点标记`: Prefer business-relevant dimensions like value, quality, feature requests, competitor mentions.
- `评论链接网址`: Must point back to the original review.
- `评论时间`: Fill only from explicit page content.
- `评论点赞数`: Fill only from explicit page content.

## Default Categories

- `Praise on product`
- `Questions/Worrying`
- `Negative emotions`
- `Suggestion`
- `Nothing particular`
- `Being supportive to the brand`
- `Expecting`
- `Being Sarcastic`
- `Competitor comparison`
- `Status update`

## Focus Mark Control Words

The analyzer normalizes model output into these dimensions:

- `value_for_money`
- `quality`
- `feature_suggestion`
- `competitor_model`
- `software_setup`
- `installation_fit`
- `brand_after_sales`
- `logistics_packaging`
- `accessory_compatibility`


path: references/setup.md
content: # Setup

## What This Skill Needs

This skill is designed for a two-phase workflow:

1. Use scripts to collect Amazon reviews and fill factual columns.
2. Use the model directly in the chat to tag the collected rows.

Keyword expansion is now manual. The normal path is `intake -> coverage-check -> decide whether to run --keywords`.
The built-in presets are `generic`, `electronics`, and `dashcam`; default keyword reuse skips previously productive keywords and temporarily suppresses recent zero-result retries.
The tuned state file lives at `<output-dir>/keyword_tuning_state.json` by default and can be rebuilt from the SQLite cache plus old keyword JSON reports with `keyword-autotune`.

The scripts do not require any external LLM API.

They only need:

1. A Chrome session with Amazon login available on `localhost:9222`
2. Python packages for scraping and workbook writing
3. Optional DeepLX access if you want machine translation before tagging

## Python Packages

Install these packages in the environment that will run the skill:

```bash
pip install pandas openpyxl requests websocket-client
```

## Chrome Remote Debugging

The script reads Amazon review pages through an already logged-in Chrome profile.

### Windows

1. Close existing Chrome windows.
2. Launch Chrome with remote debugging and your normal profile:

```powershell
"$env:ProgramFiles\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222 --user-data-dir="$env:LOCALAPPDATA\Google\Chrome\User Data"
```

If Chrome is installed under `Program Files (x86)`, adjust the path.

### macOS

```bash
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222 --user-data-dir="$HOME/Library/Application Support/Google/Chrome"
```

### Linux

```bash
google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.config/google-chrome"
```

After Chrome starts, verify that Amazon is logged in inside that same browser session.

## DeepLX Translation

If you want to fill `评论中文版` automatically before tagging, configure:

- `DEEPLX_API_URL`
- optional `DEEPLX_API_KEY`

You can place them in either:

- the current shell environment
- a local `.env` file in the working directory
- a local `.env` file in the skill root

The repo includes `.env.example` as a template. Keep the real `.env` out of Git.

Examples:

```powershell
$env:DEEPLX_API_URL="https://your-deeplx-host/translate"
$env:DEEPLX_API_KEY="your-key"
```

```bash
export DEEPLX_API_URL="https://your-deeplx-host/translate"
export DEEPLX_API_KEY="your-key"
```

## Sanity Check

Run this before the first real scrape:

```bash
python scripts/amazon_review_workbook.py doctor --url "<amazon-url>"
```

Expected result:

- URL parses to a valid `asin`
- `chrome_debug_ready` is `true`
- `deeplx_env_ready` is `true` if you plan to use `translate`
- `deeplx_reachable` is `true` if DeepLX is actually reachable

If `deeplx_reachable` is `false`, the skill can still continue:

- `translate` will preserve the factual rows and print `translation_mode=model_fallback`
- the model should then fill `评论中文版` during the tagging phase


path: references/tagging-guidelines.md
content: # Tagging Guidelines

Use this guide after `prepare-tagging`.

`prepare-tagging` produces the cropped machine prompt payload for the model. It is not the final workbook and it is not the merge source. The model should read the pending-row payload, fill only those rows, and write a separate labels JSON that `merge-build` can merge back into the full base JSON.

Before spending more requests on keyword expansion, prefer running:

```bash
python scripts/amazon_review_workbook.py coverage-check --url "<amazon-url>" --db-path "<out-dir>/amazon_review_cache.sqlite3"
```

If `coverage_ratio` is already high enough for the job, skip `--keywords`.

When you do need a keyword pass:

- prefer `--keyword-profile generic` for broad consumer products
- prefer `--keyword-profile electronics` for devices with apps/setup/accessories
- prefer `--keyword-profile dashcam` for our core dashcam scenario
- keep the default `--keyword-reuse-scope successful` so already-productive keywords are not rerun

To rebuild the tuned ordering from old experiments, run:

```bash
python scripts/amazon_review_workbook.py keyword-autotune --output-dir "<out-dir>" --db-path "<out-dir>/amazon_review_cache.sqlite3" --report-glob "<path-to-old-reports>/*keywords*.json"
```

## Columns The Model Should Fill

- `评论中文版` if still blank
- `评论概括`
- `情感倾向`
- `类别分类`
- `标签`
- `重点标记`

## Rules

- `评论中文版`: faithful translation; do not add new meaning
- `评论概括`: one sentence, focus on the main user point
- `情感倾向`: only `Positive`, `Negative`, or `Neutral`
- `类别分类`: choose 1-2 categories, join with ` / `
- `标签`: 1-3 short Chinese tags, ideally <= 10 chars each, join with `，`
- `重点标记`: prefer business-useful dimensions, join with `，`

## Allowed Categories

- `Praise on product`
- `Questions/Worrying`
- `Negative emotions`
- `Suggestion`
- `Nothing particular`
- `Being supportive to the brand`
- `Expecting`
- `Being Sarcastic`
- `Competitor comparison`
- `Status update`

## Preferred Focus Dimensions

- `性价比`
- `质量`
- `功能改进建议`
- `竞品/型号`
- `软件/设置`
- `安装/适配`
- `品牌/售后`
- `物流/包装`
- `配件/兼容性`

## Quality Gates

- Do not invent country, rating, review time, or helpful votes.
- Do not rewrite the review into marketing copy.
- Keep tags merged and stable; avoid synonym explosion.
- If a review is mostly descriptive and low-signal, `类别分类` can be `Nothing particular`.
- If a row is ambiguous, prefer conservative labels over over-claiming.

## Faster Workflow

### 1. Build canonical tags first

Before labeling a large batch, generate representative samples:

```bash
python scripts/amazon_review_workbook.py taxonomy-bootstrap --input-json "<translated-or-factual-json>" --output-dir "<out-dir>"
```

Then ask the model to read that bootstrap file and produce a short `canonical_tags.json`.

### 2. Prepare a lightweight tagging payload

```bash
python scripts/amazon_review_workbook.py prepare-tagging --input-json "<translated-or-factual-json>" --output-dir "<out-dir>" --canonical-tags-json "<out-dir>/canonical_tags.json"
```

This payload:

- strips the model input down to the minimum fields
- skips rows already found in cache
- pre-fills some heuristic sentiment/category/focus fields
- includes fixed category and focus enumerations

### 3. Let the model label only pending rows

Use the generated `*_tagging_input.json`, not the full workbook rows.

Model output should be a JSON array or an object with `items`, where each item contains:

- `seq` or `review_id`
- `评论中文版` if still blank
- `评论概括`
- `情感倾向`
- `类别分类`
- `标签`
- `重点标记`

### 4. Merge labels and update cache

```bash
python scripts/amazon_review_workbook.py merge-build --base-json "<translated-or-factual-json>" --labels-json "<model-output-json>" --output-dir "<out-dir>" --taxonomy-version "v1" --strict
```

This command:

- merges labeled rows back into the full base JSON
- reuses cached labels for identical reviews
- writes new labels into the cache
- exports the final workbook


path: scripts/amazon_review_workbook.py
content: from __future__ import annotations

import argparse
import glob
import json
import logging
import os
import re
import socket
import sqlite3
import sys
import time
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from urllib.parse import quote, urlparse

import requests
import websocket

SCRIPT_DIR = Path(__file__).resolve().parent
if str(SCRIPT_DIR) not in sys.path:
    sys.path.insert(0, str(SCRIPT_DIR))

from review_cache import (
    DEFAULT_DB_PATH,
    cleanup_stale_jobs,
    create_job,
    ensure_db,
    export_analysis_to_records,
    fetch_cached_analysis_map,
    fetch_cached_reviews,
    finish_job,
    get_cached_review_count,
    get_known_review_ids,
    get_keyword_history_map,
    get_searched_keywords,
    record_keyword_search,
    upsert_analysis_records,
    upsert_reviews,
)
from deeplx_translate import (
    probe_deeplx,
    read_env_value,
    resolve_api_key,
    resolve_api_url,
    translate_text,
    translate_texts_batch,
)
from label_workflow import (
    build_taxonomy_bootstrap,
    default_cache_path,
    load_cache,
    load_canonical_tags,
    merge_records_with_labels,
    prepare_tagging_payload,
    save_cache,
)
from review_delivery_schema import (
    build_review_link,
    normalize_helpful_votes,
    normalize_review_time,
    normalize_space,
    write_delivery_artifacts,
)

DEFAULT_OUTPUT_DIR = Path.cwd() / "amazon-review-output"
# Multi-combo strategy to bypass Amazon's 100-review-per-filter limit
# Each combo returns up to ~100 reviews; combining all combos yields all unique reviews
DEFAULT_COMBOS = [
    ("recent_all", "sortBy=recent"),
    ("helpful_all", "sortBy=helpful"),
    ("recent_5", "filterByStar=five_star&reviewerType=all_reviews&sortBy=recent"),
    ("helpful_5", "filterByStar=five_star&reviewerType=all_reviews&sortBy=helpful"),
    ("recent_4", "filterByStar=four_star&reviewerType=all_reviews&sortBy=recent"),
    ("helpful_4", "filterByStar=four_star&reviewerType=all_reviews&sortBy=helpful"),
    ("recent_3", "filterByStar=three_star&reviewerType=all_reviews&sortBy=recent"),
    ("helpful_3", "filterByStar=three_star&reviewerType=all_reviews&sortBy=helpful"),
    ("recent_2", "filterByStar=two_star&reviewerType=all_reviews&sortBy=recent"),
    ("helpful_2", "filterByStar=two_star&reviewerType=all_reviews&sortBy=helpful"),
    ("recent_1", "filterByStar=one_star&reviewerType=all_reviews&sortBy=recent"),
    ("helpful_1", "filterByStar=one_star&reviewerType=all_reviews&sortBy=helpful"),
    (
        "recent_media",
        "mediaType=media_reviews_only&reviewerType=all_reviews&sortBy=recent",
    ),
    (
        "helpful_media",
        "mediaType=media_reviews_only&reviewerType=all_reviews&sortBy=helpful",
    ),
    (
        "recent_5_avp",
        "filterByStar=five_star&reviewerType=avp_only_reviews&sortBy=recent",
    ),
    (
        "helpful_5_avp",
        "filterByStar=five_star&reviewerType=avp_only_reviews&sortBy=helpful",
    ),
    (
        "recent_current",
        "formatType=current_format&reviewerType=all_reviews&sortBy=recent",
    ),
    (
        "helpful_current",
        "formatType=current_format&reviewerType=all_reviews&sortBy=helpful",
    ),
]

FAST_COMBOS = [
    ("recent_all", "sortBy=recent"),
    ("helpful_all", "sortBy=helpful"),
]

FAST_KEYWORDS: list[str] = []

# Keyword presets are layered:
# - universal: broadly useful across many consumer products
# - electronics: common hardware/app/setup issues
# - dashcam: scenario-specific terms from our core delivery context
KEYWORD_LIBRARY = {
    "universal": [
        "quality",
        "problem",
        "broken",
        "refund",
        "return",
        "support",
        "durable",
        "price",
        "worth",
        "packaging",
        "recommend",
    ],
    "electronics": [
        "app",
        "setup",
        "install",
        "battery",
        "cable",
        "screen",
        "update",
        "firmware",
    ],
    "dashcam": [
        "video",
        "night",
        "parking",
        "gps",
        "wifi",
        "recording",
        "mount",
        "rear camera",
    ],
}

KEYWORD_PROFILES = {
    "generic": ["universal"],
    "electronics": ["universal", "electronics"],
    "dashcam": ["universal", "electronics", "dashcam"],
}

DEFAULT_KEYWORD_PROFILE = "electronics"

DEFAULT_MAX_PAGES = 10
DEFAULT_CDP_PORT = 9222
DEFAULT_COMBO_DELAY_SECONDS = 2.5
DEFAULT_KEYWORD_REUSE_SCOPE = "successful"
DEFAULT_ZERO_RESULT_RETRY_HOURS = 72.0
DEFAULT_KEYWORD_TUNING_TOP_K = 12

logger = logging.getLogger("amazon_review_workbook")

COUNTRY_MAP = {
    "united kingdom": "英国",
    "england": "英国",
    "united states": "美国",
    "usa": "美国",
    "italia": "意大利",
    "italy": "意大利",
    "germania": "德国",
    "germany": "德国",
    "france": "法国",
    "francia": "法国",
    "spain": "西班牙",
    "spagna": "西班牙",
    "canada": "加拿大",
    "japan": "日本",
}

HOST_COUNTRY_MAP = {
    "amazon.co.uk": "英国",
    "amazon.com": "美国",
    "amazon.it": "意大利",
    "amazon.de": "德国",
    "amazon.fr": "法国",
    "amazon.es": "西班牙",
    "amazon.ca": "加拿大",
    "amazon.co.jp": "日本",
}


def configure_stdio() -> None:
    for stream_name in ("stdout", "stderr"):
        stream = getattr(sys, stream_name, None)
        reconfigure = getattr(stream, "reconfigure", None)
        if callable(reconfigure):
            reconfigure(encoding="utf-8")


def parse_product_url(url: str) -> tuple[str, str]:
    parsed = urlparse(url)
    match = re.search(
        r"/(?:dp|gp/product|product-reviews)/([A-Z0-9]{10})", parsed.path, re.I
    )
    if not match:
        raise ValueError(f"无法从 URL 解析 ASIN: {url}")
    host = parsed.netloc or "www.amazon.com"
    return match.group(1).upper(), host


def build_review_url(host: str, asin: str) -> str:
    return f"https://{host}/product-reviews/{asin}/ref=cm_cr_dp_d_show_all_btm?ie=UTF8"


def build_collect_output_path(output_dir: Path, asin: str) -> Path:
    return build_machine_output_path(output_dir, asin)


def build_machine_output_path(output_dir: Path, asin: str) -> Path:
    return output_dir.expanduser() / f"amazon_{asin}_review_rows_machine.json"


def _record_keys(record: dict[str, Any]) -> list[str]:
    keys: list[str] = []
    for field in ("review_id", "seq", "序号", "position"):
        value = normalize_space(record.get(field))
        if value and value not in keys:
            keys.append(value)
    return keys


def load_layered_records(input_path: Path) -> list[dict[str, Any]]:
    records = load_json_records(input_path)
    asin = infer_asin_from_records(records, input_path)
    machine_path = build_machine_output_path(input_path.parent, asin)
    if not machine_path.exists() or machine_path.resolve() == input_path.resolve():
        return records

    machine_records = load_json_records(machine_path)
    supplemental_by_key: dict[str, dict[str, Any]] = {}
    for record in records:
        for key in _record_keys(record):
            supplemental_by_key.setdefault(key, record)

    layered: list[dict[str, Any]] = []
    for record in machine_records:
        row = dict(record)
        supplemental = None
        for key in _record_keys(record):
            supplemental = supplemental_by_key.get(key)
            if supplemental:
                break
        if supplemental:
            for field, value in supplemental.items():
                if normalize_space(value):
                    row[field] = value
        layered.append(row)
    return layered


def build_factual_output_paths(output_dir: Path, asin: str) -> tuple[Path, Path, Path]:
    base = output_dir.expanduser() / f"amazon_{asin}_review_rows_factual"
    return (
        base.with_suffix(".json"),
        base.with_suffix(".xlsx"),
        base.with_suffix(".csv"),
    )


def build_translated_output_paths(
    output_dir: Path, asin: str
) -> tuple[Path, Path, Path]:
    base = output_dir.expanduser() / f"amazon_{asin}_review_rows_translated"
    return (
        base.with_suffix(".json"),
        base.with_suffix(".xlsx"),
        base.with_suffix(".csv"),
    )


def build_final_output_paths(output_dir: Path, stem: str) -> tuple[Path, Path, Path]:
    base = output_dir.expanduser() / stem
    return (
        base.with_suffix(".json"),
        base.with_suffix(".xlsx"),
        base.with_suffix(".csv"),
    )


def build_prepare_output_path(output_dir: Path, asin: str) -> Path:
    return output_dir.expanduser() / f"amazon_{asin}_tagging_input.json"


def build_bootstrap_output_path(output_dir: Path, asin: str) -> Path:
    return output_dir.expanduser() / f"amazon_{asin}_taxonomy_bootstrap.json"


def default_keyword_tuning_state_path(output_dir: Path) -> Path:
    return output_dir.expanduser() / "keyword_tuning_state.json"


def dedupe_preserve_order(items: list[str]) -> list[str]:
    seen: set[str] = set()
    result: list[str] = []
    for item in items:
        if item not in seen:
            seen.add(item)
            result.append(item)
    return result


def build_keyword_profile(profile: str) -> list[str]:
    layers = KEYWORD_PROFILES.get(profile, KEYWORD_PROFILES[DEFAULT_KEYWORD_PROFILE])
    combined: list[str] = []
    for layer in layers:
        combined.extend(KEYWORD_LIBRARY.get(layer, []))
    return dedupe_preserve_order(combined)


DEFAULT_KEYWORDS = build_keyword_profile(DEFAULT_KEYWORD_PROFILE)


def load_keyword_tuning_state(path: Path | None) -> dict[str, Any]:
    if path is None or not path.exists():
        return {}
    try:
        payload = json.loads(path.read_text(encoding="utf-8"))
    except json.JSONDecodeError:
        return {}
    return payload if isinstance(payload, dict) else {}


def score_keyword_stats(stats: dict[str, Any]) -> float:
    best_new = float(stats.get("best_new_count") or 0)
    total_new = float(stats.get("total_new_count") or 0)
    positive_runs = float(stats.get("positive_runs") or 0)
    zero_runs = float(stats.get("zero_runs") or 0)
    total_runs = max(float(stats.get("total_runs") or 0), 1.0)
    hit_rate = positive_runs / total_runs
    return round(best_new * 5 + total_new * 2 + hit_rate * 10 - zero_runs * 1.5, 3)


def build_recommended_keywords_from_stats(
    profile: str,
    keyword_stats: dict[str, dict[str, Any]],
    *,
    top_k: int = DEFAULT_KEYWORD_TUNING_TOP_K,
) -> list[str]:
    base_keywords = build_keyword_profile(profile)
    ranked: list[tuple[float, int, int, str]] = []
    for base_index, keyword in enumerate(base_keywords):
        stats = keyword_stats.get(keyword, {})
        score = score_keyword_stats(stats)
        positive_runs = int(stats.get("positive_runs") or 0)
        total_new = int(stats.get("total_new_count") or 0)
        ranked.append((score, positive_runs, total_new, keyword))

    ranked.sort(key=lambda item: (-item[0], -item[1], -item[2], base_keywords.index(item[3])))
    positives = [item[3] for item in ranked if item[1] > 0]
    unexplored = [item[3] for item in ranked if item[1] == 0 and item[0] == 0]
    weak = [item[3] for item in ranked if item[1] == 0 and item[0] != 0]
    ordered = dedupe_preserve_order(positives + unexplored + weak)
    if top_k > 0:
        return ordered[:top_k]
    return ordered


def get_tuned_keywords(
    profile: str,
    *,
    tuning_state: dict[str, Any],
    top_k: int = DEFAULT_KEYWORD_TUNING_TOP_K,
) -> list[str]:
    profile_payload = (tuning_state.get("profiles") or {}).get(profile)
    if not isinstance(profile_payload, dict):
        return build_keyword_profile(profile)[:top_k]
    keywords = profile_payload.get("recommended_keywords") or []
    cleaned = dedupe_preserve_order(
        [normalize_space(item) for item in keywords if normalize_space(item)]
    )
    if cleaned:
        return cleaned[:top_k] if top_k > 0 else cleaned
    return build_keyword_profile(profile)[:top_k]


def resolve_keyword_plan(
    keywords: list[str] | None,
    keyword_profile: str,
    *,
    tuning_state: dict[str, Any] | None = None,
    top_k: int = DEFAULT_KEYWORD_TUNING_TOP_K,
) -> tuple[list[str], str]:
    if keywords is None:
        return [], "off"
    cleaned = dedupe_preserve_order(
        [normalize_space(item) for item in keywords if normalize_space(item)]
    )
    if cleaned:
        return cleaned, "custom"
    tuned = get_tuned_keywords(
        keyword_profile, tuning_state=tuning_state or {}, top_k=top_k
    )
    return tuned, f"profile:{keyword_profile}:tuned"


def parse_utc_timestamp(value: Any) -> datetime | None:
    text = normalize_space(value)
    if not text:
        return None
    for fmt in ("%Y-%m-%dT%H:%M:%S.%fZ", "%Y-%m-%dT%H:%M:%SZ"):
        try:
            return datetime.strptime(text, fmt).replace(tzinfo=timezone.utc)
        except ValueError:
            continue
    return None


def should_skip_keyword(
    history_entry: dict[str, Any] | None,
    *,
    reuse_scope: str,
    zero_result_retry_hours: float,
    now_utc: datetime | None = None,
) -> tuple[bool, str]:
    if not history_entry:
        return False, ""
    if reuse_scope == "none":
        return False, ""

    best_new_count = int(history_entry.get("best_new_count") or 0)
    if reuse_scope in {"successful", "all"} and best_new_count > 0:
        return True, "successful_history"

    if reuse_scope == "all":
        return True, "history"

    if zero_result_retry_hours <= 0:
        return False, ""

    searched_at = parse_utc_timestamp(history_entry.get("searched_at"))
    if not searched_at:
        return False, ""

    now = now_utc or datetime.now(timezone.utc)
    age_hours = (now - searched_at).total_seconds() / 3600
    if age_hours < zero_result_retry_hours:
        return True, "recent_zero_history"
    return False, ""


def merge_keyword_observation(
    bucket: dict[str, dict[str, Any]], keyword: str, new_count: int, source: str
) -> None:
    keyword = normalize_space(keyword)
    if not keyword:
        return
    stats = bucket.setdefault(
        keyword,
        {
            "keyword": keyword,
            "total_runs": 0,
            "positive_runs": 0,
            "zero_runs": 0,
            "total_new_count": 0,
            "best_new_count": 0,
            "sources": [],
        },
    )
    stats["total_runs"] += 1
    stats["total_new_count"] += int(new_count)
    stats["best_new_count"] = max(int(stats["best_new_count"]), int(new_count))
    if int(new_count) > 0:
        stats["positive_runs"] += 1
    else:
        stats["zero_runs"] += 1
    if source not in stats["sources"]:
        stats["sources"].append(source)


def aggregate_keyword_stats_from_db(db_path: Path) -> dict[str, dict[str, Any]]:
    if not db_path.exists():
        return {}
    conn = ensure_db(db_path)
    rows = conn.execute(
        """SELECT keyword, new_count, best_new_count, search_count
           FROM keyword_history"""
    ).fetchall()
    conn.close()
    result: dict[str, dict[str, Any]] = {}
    for keyword, new_count, best_new_count, search_count in rows:
        merge_keyword_observation(result, str(keyword), int(best_new_count or new_count or 0), "db_best")
        stats = result[str(keyword)]
        stats["total_runs"] = max(int(stats["total_runs"]), int(search_count or 1))
        stats["best_new_count"] = max(int(stats["best_new_count"]), int(best_new_count or 0))
        stats["total_new_count"] = max(int(stats["total_new_count"]), int(best_new_count or new_count or 0))
    return result


def aggregate_keyword_stats_from_reports(report_paths: list[Path]) -> dict[str, dict[str, Any]]:
    result: dict[str, dict[str, Any]] = {}
    for path in report_paths:
        if not path.exists():
            continue
        try:
            payload = json.loads(path.read_text(encoding="utf-8"))
        except json.JSONDecodeError:
            continue
        if not isinstance(payload, dict):
            continue
        stats_rows = payload.get("stats")
        if not isinstance(stats_rows, list):
            continue
        for row in stats_rows:
            if not isinstance(row, dict):
                continue
            keyword = normalize_space(row.get("kw") or row.get("keyword"))
            new_count = int(row.get("added_new") or row.get("new") or 0)
            merge_keyword_observation(result, keyword, new_count, path.name)
    return result


def combine_keyword_stats(*sources: dict[str, dict[str, Any]]) -> dict[str, dict[str, Any]]:
    combined: dict[str, dict[str, Any]] = {}
    for source in sources:
        for keyword, stats in source.items():
            entry = combined.setdefault(
                keyword,
                {
                    "keyword": keyword,
                    "total_runs": 0,
                    "positive_runs": 0,
                    "zero_runs": 0,
                    "total_new_count": 0,
                    "best_new_count": 0,
                    "sources": [],
                },
            )
            entry["total_runs"] += int(stats.get("total_runs") or 0)
            entry["positive_runs"] += int(stats.get("positive_runs") or 0)
            entry["zero_runs"] += int(stats.get("zero_runs") or 0)
            entry["total_new_count"] += int(stats.get("total_new_count") or 0)
            entry["best_new_count"] = max(
                int(entry["best_new_count"]), int(stats.get("best_new_count") or 0)
            )
            entry["sources"] = dedupe_preserve_order(
                list(entry["sources"]) + list(stats.get("sources") or [])
            )
    for entry in combined.values():
        entry["score"] = score_keyword_stats(entry)
    return combined


def build_keyword_tuning_state(
    *,
    keyword_stats: dict[str, dict[str, Any]],
    top_k: int = DEFAULT_KEYWORD_TUNING_TOP_K,
) -> dict[str, Any]:
    profiles: dict[str, Any] = {}
    for profile in sorted(KEYWORD_PROFILES):
        recommended = build_recommended_keywords_from_stats(
            profile, keyword_stats, top_k=top_k
        )
        profiles[profile] = {
            "recommended_keywords": recommended,
            "base_keywords": build_keyword_profile(profile),
        }
    global_top = sorted(
        keyword_stats.values(),
        key=lambda item: (-float(item.get("score") or 0), item["keyword"]),
    )
    return {
        "metadata": {
            "generated_at": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ"),
            "keyword_count": len(keyword_stats),
            "top_k": top_k,
        },
        "profiles": profiles,
        "global_keyword_stats": global_top,
    }


def refresh_keyword_tuning_state(
    *,
    db_path: Path,
    output_path: Path,
    report_paths: list[Path] | None = None,
    top_k: int = DEFAULT_KEYWORD_TUNING_TOP_K,
) -> dict[str, Any]:
    db_stats = aggregate_keyword_stats_from_db(db_path)
    report_stats = aggregate_keyword_stats_from_reports(report_paths or [])
    combined = combine_keyword_stats(db_stats, report_stats)
    payload = build_keyword_tuning_state(keyword_stats=combined, top_k=top_k)
    output_path.parent.mkdir(parents=True, exist_ok=True)
    output_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
    return payload


def parse_count_token(token: str) -> int | None:
    digits = re.sub(r"[^\d]", "", token)
    if not digits:
        return None
    try:
        return int(digits)
    except ValueError:
        return None


def extract_named_count(
    texts: list[str], patterns: list[str]
) -> tuple[int | None, str]:
    for text in texts:
        collapsed = normalize_space(text)
        lowered = collapsed.lower()
        for pattern in patterns:
            match = re.search(pattern, lowered, re.I)
            if not match:
                continue
            value = parse_count_token(match.group(1))
            if value is not None:
                return value, collapsed
    return None, ""


def extract_page_totals(texts: list[str]) -> dict[str, Any]:
    review_patterns = [
        r"(?:showing|show)\s+\d+\s*-\s*\d+\s+(?:of|sur|von|de)\s+(\d[\d.,\s]*)\s+(?:customer\s+)?(?:global\s+)?(?:reviews?|recensioni|reseñas|avis|bewertungen)",
        r"(\d[\d.,\s]*)\s+(?:customer\s+)?(?:global\s+)?(?:reviews?|recensioni|reseñas|avis|bewertungen)",
    ]
    rating_patterns = [
        r"(?:showing|show)\s+\d+\s*-\s*\d+\s+(?:of|sur|von|de)\s+(\d[\d.,\s]*)\s+(?:global\s+)?(?:ratings|valutazioni|calificaciones|évaluations)",
        r"(\d[\d.,\s]*)\s+(?:global\s+)?(?:ratings|valutazioni|calificaciones|évaluations)",
    ]
    review_count, review_evidence = extract_named_count(texts, review_patterns)
    rating_count, rating_evidence = extract_named_count(texts, rating_patterns)
    return {
        "page_total_reviews": review_count,
        "page_total_reviews_evidence": review_evidence,
        "page_total_ratings": rating_count,
        "page_total_ratings_evidence": rating_evidence,
    }


def port_is_open(host: str, port: int) -> bool:
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
        sock.settimeout(1.5)
        return sock.connect_ex((host, port)) == 0


@dataclass
class PageSnapshot:
    href: str
    title: str
    next_href: str | None
    reviews: list[dict[str, Any]]


class BrowserSession:
    def __init__(self, port: int) -> None:
        self.port = port
        self.ws_url: str | None = None

    @property
    def http_base(self) -> str:
        return f"http://127.0.0.1:{self.port}"

    def wait_until_ready(self, timeout_seconds: int = 15) -> None:
        deadline = time.time() + timeout_seconds
        last_error = "Chrome remote debugging not ready"
        while time.time() < deadline:
            try:
                targets = requests.get(f"{self.http_base}/json", timeout=5).json()
                if not targets:
                    targets = [
                        requests.put(
                            f"{self.http_base}/json/new?about:blank", timeout=5
                        ).json()
                    ]
                pages = [
                    item
                    for item in targets
                    if item.get("type") == "page" and item.get("webSocketDebuggerUrl")
                ]
                if pages:
                    self.ws_url = str(pages[0]["webSocketDebuggerUrl"])
                    return
                last_error = "no page targets"
            except Exception as exc:  # noqa: BLE001
                last_error = str(exc)
            time.sleep(1)
        raise RuntimeError(last_error)

    def send(
        self, method: str, params: dict[str, Any] | None = None, *, timeout: int = 30
    ) -> Any:
        if not self.ws_url:
            raise RuntimeError("Browser target is not initialized")
        ws = websocket.create_connection(
            self.ws_url, suppress_origin=True, timeout=timeout
        )
        try:
            ws.send(
                json.dumps(
                    {
                        "id": 1,
                        "method": method,
                        "params": params or {},
                    }
                )
            )
            deadline = time.time() + timeout
            max_attempts = 200
            attempts = 0
            while time.time() < deadline and attempts < max_attempts:
                attempts += 1
                message = json.loads(ws.recv())
                if message.get("id") != 1:
                    continue
                result = message.get("result", {})
                if message.get("error"):
                    raise RuntimeError(
                        message["error"].get("message") or f"CDP {method} failed"
                    )
                return result
            raise RuntimeError(
                f"CDP {method} timed out after {timeout}s / {max_attempts} attempts"
            )
        finally:
            ws.close()

    def eval(self, expression: str, *, timeout: int = 30) -> Any:
        result = self.send(
            "Runtime.evaluate",
            {
                "expression": expression,
                "returnByValue": True,
                "awaitPromise": True,
            },
            timeout=timeout,
        )
        if result.get("exceptionDetails"):
            raise RuntimeError(
                result["exceptionDetails"].get("text") or "CDP evaluate failed"
            )
        return result.get("result", {}).get("value")

    def navigate(self, url: str) -> None:
        self.send("Page.navigate", {"url": url}, timeout=20)
        time.sleep(4)


def extract_page(browser: BrowserSession) -> PageSnapshot:
    expression = r"""
(() => {
  const normalize = (value) => (value || '').replace(/\s+/g, ' ').trim();
  // Use textContent from cloned DOM to bypass Edge/Chrome auto-translation
  // which only modifies the rendered innerText, not the original DOM text.
  const moreReviews = Array.from(document.querySelectorAll('a, button, input[type="button"], input[type="submit"]')).find((el) => {
    const label = ((el.innerText || el.value || el.getAttribute('aria-label') || '') + '').replace(/\s+/g, ' ').trim();
    return /^(show|load)\s+\d+\s+more reviews$/i.test(label) || /^(show|load)\s+more reviews$/i.test(label);
  });
  const reviews = Array.from(document.querySelectorAll('[data-hook="review"]')).map((el, index) => {
    const rawText = (selector) => {
      const node = el.querySelector(selector);
      if (!node) return '';
      const clone = node.cloneNode(true);
      const translateEls = clone.querySelectorAll('[translate="no"], [class*="translated"]');
      translateEls.forEach(e => e.remove());
      return normalize(clone.textContent || '');
    };
    const reviewEl = el;
    let reviewId = reviewEl.id || '';
    if (!reviewId) {
      const titleLink = reviewEl.querySelector('a[data-hook="review-title"]');
      if (titleLink && titleLink.href) {
        const m = titleLink.href.match(/\/R([A-Z0-9]+)/i);
        if (m) reviewId = 'R' + m[1];
      }
    }
    if (!reviewId) {
      const dataId = reviewEl.getAttribute('data-review-id') || reviewEl.getAttribute('data-contentid');
      if (dataId) reviewId = dataId;
    }
    return {
      position: index + 1,
      review_id: reviewId,
      title: rawText('[data-hook="review-title"]'),
      rating_text: rawText('[data-hook="review-star-rating"], [data-hook="cmps-review-star-rating"]'),
      country_date: rawText('[data-hook="review-date"]'),
      body: rawText('[data-hook="review-body"]'),
      author: rawText('.a-profile-name'),
      helpful_votes: rawText('[data-hook="helpful-vote-statement"], .cr-vote-text'),
      review_link: el.querySelector('a[data-hook="review-title"]')?.href || ''
    };
  });
  return {
    href: location.href,
    title: document.title,
    next_href: (moreReviews && (moreReviews.href || moreReviews.getAttribute('href'))) || document.querySelector('.a-pagination .a-last a')?.href || null,
    reviews
  };
})()
"""
    payload = browser.eval(expression)
    return PageSnapshot(
        href=str(payload.get("href", "")),
        title=str(payload.get("title", "")),
        next_href=str(payload["next_href"]) if payload.get("next_href") else None,
        reviews=list(payload.get("reviews") or []),
    )


def wait_loaded(browser: BrowserSession, timeout_seconds: int = 25) -> PageSnapshot:
    deadline = time.time() + timeout_seconds
    latest = extract_page(browser)
    while time.time() < deadline:
        if latest.reviews:
            return latest
        time.sleep(1)
        latest = extract_page(browser)
    return latest


def click_next(browser: BrowserSession) -> bool:
    expression = r"""
(() => {
  const candidates = Array.from(document.querySelectorAll('a, button, input[type="button"], input[type="submit"]'));
  const target = candidates.find((el) => {
    const label = ((el.innerText || el.value || el.getAttribute('aria-label') || '') + '').replace(/\s+/g, ' ').trim();
    return /^(show|load)\s+\d+\s+more reviews$/i.test(label) || /^(show|load)\s+more reviews$/i.test(label);
  }) || document.querySelector('.a-pagination .a-last a');
  if (!target) return false;
  target.click();
  return true;
})()
"""
    return bool(browser.eval(expression))


def wait_for_page_change(
    browser: BrowserSession, previous_snapshot: PageSnapshot, timeout_seconds: int = 20
) -> PageSnapshot:
    deadline = time.time() + timeout_seconds
    latest = extract_page(browser)
    while time.time() < deadline:
        reviews = latest.reviews
        first_id = reviews[0].get("review_id") if reviews else None
        last_id = reviews[-1].get("review_id") if reviews else None
        if (
            len(reviews) != len(previous_snapshot.reviews)
            or first_id
            != (
                previous_snapshot.reviews[0].get("review_id")
                if previous_snapshot.reviews
                else None
            )
            or last_id
            != (
                previous_snapshot.reviews[-1].get("review_id")
                if previous_snapshot.reviews
                else None
            )
        ):
            return latest
        time.sleep(2)
        latest = extract_page(browser)
    return latest


def fetch_review_page_totals(review_url: str, port: int) -> dict[str, Any]:
    browser = BrowserSession(port)
    browser.wait_until_ready()
    browser.navigate(review_url)
    _enforce_notranslate(browser)
    payload = browser.eval(
        r"""
(() => {
  const selectors = [
    '[data-hook="cr-filter-info-review-rating-count"]',
    '#filter-info-section',
    '#acrCustomerReviewText',
    '[data-hook="total-review-count"]',
    '.cr-filter-info-review-rating-count',
    '[data-hook="review"] [data-hook="review-date"]'
  ];
  const texts = [];
  for (const selector of selectors) {
    for (const node of document.querySelectorAll(selector)) {
      const text = (node.textContent || '').replace(/\s+/g, ' ').trim();
      if (text) texts.push(text);
    }
  }
  return {
    href: location.href,
    title: document.title,
    texts: Array.from(new Set(texts)).slice(0, 20)
  };
})()
"""
    )
    texts = [str(item) for item in payload.get("texts") or [] if normalize_space(item)]
    totals = extract_page_totals(texts)
    return {
        "probe_url": str(payload.get("href", review_url)),
        "probe_title": str(payload.get("title", "")),
        "probe_texts": texts,
        **totals,
    }


def infer_country(country_date: str, host: str) -> str:
    lowered = normalize_space(country_date).lower()
    for key, value in COUNTRY_MAP.items():
        if key in lowered:
            return value
    for key, value in HOST_COUNTRY_MAP.items():
        if key in host:
            return value
    return ""


def parse_rating(rating_text: str) -> str:
    match = re.search(r"(\d+(?:[.,]\d+)?)", str(rating_text))
    if not match:
        return ""
    value = match.group(1).replace(",", ".")
    if value.endswith(".0"):
        value = value[:-2]
    return value


def strip_rating_prefix(title: str) -> str:
    value = normalize_space(title)
    patterns = [
        r"^\s*\d+(?:[.,]\d+)?\s+out of\s+\d+\s+stars\s*",
        r"^\s*\d+(?:[.,]\d+)?\s+su\s+\d+\s+stelle\s*",
        r"^\s*\d+(?:[.,]\d+)?\s+de\s+\d+\s+estrellas\s*",
        r"^\s*\d+(?:[.,]\d+)?\s+sur\s+\d+\s+étoiles\s*",
    ]
    for pattern in patterns:
        value = re.sub(pattern, "", value, flags=re.I)
    return normalize_space(value)


def merge_review_row(target: dict[str, Any], source: dict[str, Any]) -> None:
    """Fill gaps in target from source. First-wins, fill-gaps strategy."""
    for key, value in source.items():
        if value in (None, "", []):
            continue
        if not target.get(key):
            target[key] = value


def _enforce_notranslate(browser: BrowserSession) -> None:
    """Disable Edge/Chrome auto-translation and reload page."""
    browser.eval("""
(() => {
  let translateMeta = document.querySelector('meta[name="google"]');
  if (!translateMeta) {
    const m = document.createElement('meta');
    m.name = 'google';
    m.content = 'notranslate';
    document.head.appendChild(m);
  }
  document.documentElement.setAttribute('translate', 'no');
  const htmlEl = document.documentElement;
  if (htmlEl.classList.contains('translated-yes')) {
    htmlEl.classList.remove('translated-yes');
    htmlEl.classList.add('translated-no');
  }
  if (document.body && document.body.classList.contains('translated-yes')) {
    document.body.classList.remove('translated-yes');
    document.body.classList.add('translated-no');
  }
  document.querySelectorAll('.goog-te-banner-frame, .goog-te-menu-frame, .skiptranslate, .translated-text').forEach(el => el.remove());
  return 'notranslate enforced';
})()
""")
    time.sleep(1)
    browser.eval("window.location.reload()")
    time.sleep(5)


def _collect_single_combo(
    browser: BrowserSession,
    combo_url: str,
    combo_name: str,
    *,
    max_pages: int,
    resume_from_page: int,
    host: str,
    asin: str,
    seen_review_ids: set[str],
    all_rows: list[dict[str, Any]],
) -> int:
    """Collect reviews from a single filter/sort combo. Returns new review count."""
    browser.navigate(combo_url)
    time.sleep(4)
    _enforce_notranslate(browser)

    page_title = browser.eval("document.title")
    if page_title and any(
        blocked in str(page_title).lower()
        for blocked in ("robot check", "captcha", "sorry", "problem loading")
    ):
        print(f"[{combo_name}] 阻断页面，跳过")
        return 0

    pages: list[dict[str, Any]] = []
    seen_signatures: set[tuple[str, str, int]] = set()
    current = wait_loaded(browser)

    if not current.reviews:
        print(f"[{combo_name}] 未检测到评论，跳过")
        return 0

    pages_to_skip = max(resume_from_page - 1, 0)
    while pages_to_skip > 0:
        if not current.next_href or not click_next(browser):
            print(f"[{combo_name}] 无法跳到第 {resume_from_page} 页，提前结束")
            return 0
        current = wait_for_page_change(browser, current)
        pages_to_skip -= 1

    start_page = max(resume_from_page, 1)
    for page_no in range(start_page, start_page + max_pages):
        if not current.reviews:
            break
        first_id = str(current.reviews[0].get("review_id") or "")
        last_id = str(current.reviews[-1].get("review_id") or "")
        signature = (first_id, last_id, len(current.reviews))
        if not first_id or signature in seen_signatures:
            break

        seen_signatures.add(signature)
        pages.append(
            {
                "page_no": page_no,
                "href": current.href,
                "title": current.title,
                "next_href": current.next_href,
                "reviews": current.reviews,
            }
        )

        if not current.next_href:
            break
        if not click_next(browser):
            break
        current = wait_for_page_change(browser, current)

    new_count = 0
    for page in pages:
        for review in page["reviews"]:
            review_id = str(review.get("review_id") or "")
            if review_id and review_id in seen_review_ids:
                continue
            if review_id:
                seen_review_ids.add(review_id)
            row = dict(review)
            row["seq"] = str(len(all_rows) + new_count + 1)
            row["page_no"] = page["page_no"]
            row["source_href"] = page["href"]
            row["host"] = host
            row["asin"] = asin
            row["review_time"] = normalize_review_time(row.get("country_date", ""))
            row["helpful_votes"] = normalize_helpful_votes(row.get("helpful_votes", ""))
            row["review_link"] = build_review_link(
                str(row.get("review_id") or ""), host, row.get("review_link")
            )
            all_rows.append(row)
            new_count += 1

    if new_count > 0:
        print(f"[{combo_name}] 新增 {new_count} 条评论（共 {len(pages)} 页）")
    else:
        print(f"[{combo_name}] 无新评论")
    return new_count


def collect_reviews(
    url: str,
    *,
    port: int,
    max_pages: int,
    resume_from: int = 0,
    mode: str = "deep",
    db_path: Path | None = None,
    refresh_cache: bool = False,
    keywords: list[str] | None = None,
    keyword_profile: str = DEFAULT_KEYWORD_PROFILE,
    keyword_reuse_scope: str = DEFAULT_KEYWORD_REUSE_SCOPE,
    zero_result_retry_hours: float = DEFAULT_ZERO_RESULT_RETRY_HOURS,
    keyword_tuning_state_path: Path | None = None,
    combo_delay_seconds: float = DEFAULT_COMBO_DELAY_SECONDS,
) -> dict[str, Any]:
    asin, host = parse_product_url(url)
    review_url = build_review_url(host, asin)

    # Setup cache
    if db_path is None:
        db_path = DEFAULT_DB_PATH
    conn = ensure_db(db_path)
    cleanup_stale_jobs(conn)

    cached_count = get_cached_review_count(conn, host, asin)
    known_ids = get_known_review_ids(conn, host, asin) if not refresh_cache else set()

    # Skip collection if cache is fresh and mode is fast
    if not refresh_cache and mode == "fast" and cached_count > 0:
        print(f"缓存已有 {cached_count} 条评论，fast 模式跳过采集。")
        cached_rows = fetch_cached_reviews(conn, host, asin)
        # Re-normalize
        for i, row in enumerate(cached_rows, start=1):
            row["seq"] = str(i)
            row["host"] = host
            row["asin"] = asin
        conn.close()
        return {
            "asin": asin,
            "host": host,
            "source_url": url,
            "review_url": review_url,
            "mode": mode,
            "combo_count": 0,
            "cached_total": cached_count,
            "row_count": len(cached_rows),
            "results": cached_rows,
        }

    # Create job
    job_id = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
    create_job(conn, job_id, host, asin, url)

    browser = BrowserSession(port)
    browser.wait_until_ready()
    tuning_state = load_keyword_tuning_state(keyword_tuning_state_path)

    # Select combo strategy
    if mode == "fast":
        combos = FAST_COMBOS
        planned_keywords = FAST_KEYWORDS
        keyword_mode = "off"
    else:
        combos = DEFAULT_COMBOS
        planned_keywords, keyword_mode = resolve_keyword_plan(
            keywords,
            keyword_profile,
            tuning_state=tuning_state,
        )

    all_rows: list[dict[str, Any]] = []
    seen_review_ids: set[str] = set(known_ids)
    stats: list[dict[str, Any]] = []

    print(
        f"开始采集评论，模式: {mode}，{len(combos)} 个筛选组合 + {len(planned_keywords)} 个关键词"
    )
    if cached_count > 0:
        print(f"缓存已有 {cached_count} 条评论，本次只采集新增的")

    # Phase 1: Star/sort combos
    for combo_name, combo_params in combos:
        combo_url = f"https://{host}/product-reviews/{asin}/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&{combo_params}"
        new_count = _collect_single_combo(
            browser,
            combo_url,
            combo_name,
            max_pages=max_pages,
            resume_from_page=resume_from,
            host=host,
            asin=asin,
            seen_review_ids=seen_review_ids,
            all_rows=all_rows,
        )
        stats.append(
            {
                "combo": combo_name,
                "new": new_count,
                "total": len(all_rows),
            }
        )
        if combo_delay_seconds > 0 and (
            combo_name != combos[-1][0] or planned_keywords
        ):
            time.sleep(combo_delay_seconds)

    # Phase 2: Keyword search with history tracking and smart skip
    keyword_history = (
        get_keyword_history_map(conn, host, asin) if not refresh_cache else {}
    )
    skipped_keywords = 0
    skipped_successful_keywords = 0
    skipped_recent_zero_keywords = 0
    consecutive_zero_keywords = 0

    for kw in planned_keywords:
        should_skip, reason = should_skip_keyword(
            keyword_history.get(kw),
            reuse_scope=keyword_reuse_scope,
            zero_result_retry_hours=zero_result_retry_hours,
        )
        if should_skip:
            skipped_keywords += 1
            if reason == "successful_history":
                skipped_successful_keywords += 1
            elif reason == "recent_zero_history":
                skipped_recent_zero_keywords += 1
            continue

        kw_name = kw.replace(" ", "_")
        kw_params = f"filterByStar=all_stars&reviewerType=all_reviews&sortBy=recent&filterByKeyword={quote(kw)}"
        combo_url = f"https://{host}/product-reviews/{asin}/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&{kw_params}"
        new_count = _collect_single_combo(
            browser,
            combo_url,
            f"kw_{kw_name}",
            max_pages=max_pages,
            resume_from_page=resume_from,
            host=host,
            asin=asin,
            seen_review_ids=seen_review_ids,
            all_rows=all_rows,
        )
        stats.append(
            {
                "combo": f"kw_{kw_name}",
                "keyword": kw,
                "new": new_count,
                "total": len(all_rows),
            }
        )
        # Record keyword search in history
        record_keyword_search(conn, host, asin, kw, job_id, new_count, len(all_rows))

        # Smart early termination:
        # - If no new reviews from last 3 keywords, stop
        # - If we have 500+ reviews, stop
        if new_count == 0:
            consecutive_zero_keywords += 1
        else:
            consecutive_zero_keywords = 0
        if len(all_rows) >= 500:
            print(f"已达到 500+ 条评论，停止关键词搜索")
            break
        if consecutive_zero_keywords >= 3:
            print("连续 3 个关键词没有新增评论，停止关键词搜索")
            break
        if combo_delay_seconds > 0 and kw != planned_keywords[-1]:
            time.sleep(combo_delay_seconds)

    if skipped_keywords > 0:
        print(f"跳过 {skipped_keywords} 个已搜索过的关键词")
    if skipped_successful_keywords > 0:
        print(f"其中 {skipped_successful_keywords} 个关键词因历史命中过结果被复用跳过")
    if skipped_recent_zero_keywords > 0:
        print(
            f"其中 {skipped_recent_zero_keywords} 个关键词因近期零结果记录被临时跳过"
        )

    # Save to cache
    new_in_session = len(all_rows)
    upsert_reviews(conn, host, asin, job_id, all_rows)
    final_cached_count = get_cached_review_count(conn, host, asin)
    finish_job(
        conn,
        job_id,
        "success",
        current_seen_count=len(all_rows),
        new_review_count=new_in_session,
        cached_total=final_cached_count,
        stats=stats,
    )

    # Load full cache for output (includes previously cached reviews)
    cached_rows = fetch_cached_reviews(conn, host, asin)
    conn.close()

    if keyword_tuning_state_path:
        try:
            refresh_keyword_tuning_state(
                db_path=db_path,
                output_path=keyword_tuning_state_path,
            )
        except Exception as exc:  # noqa: BLE001
            print(f"keyword_tuning_state_refresh_failed={exc}")

    # Re-number and normalize
    for i, row in enumerate(cached_rows, start=1):
        row["seq"] = str(i)
        row["host"] = host
        row["asin"] = asin

    return {
        "asin": asin,
        "host": host,
        "source_url": url,
        "review_url": review_url,
        "mode": mode,
        "combo_count": len(combos),
        "keyword_count": len(planned_keywords),
        "keyword_mode": keyword_mode,
        "keyword_profile": keyword_profile,
        "keyword_tuning_state_path": str(keyword_tuning_state_path)
        if keyword_tuning_state_path
        else "",
        "keyword_reuse_scope": keyword_reuse_scope,
        "skipped_keywords": skipped_keywords,
        "skipped_successful_keywords": skipped_successful_keywords,
        "skipped_recent_zero_keywords": skipped_recent_zero_keywords,
        "new_in_session": new_in_session,
        "cached_total": final_cached_count,
        "row_count": len(cached_rows),
        "stats": stats,
        "results": cached_rows,
    }


def load_json_records(path: Path) -> list[dict[str, Any]]:
    payload = json.loads(path.read_text(encoding="utf-8"))
    if isinstance(payload, list):
        return [item for item in payload if isinstance(item, dict)]
    if isinstance(payload, dict):
        for key in ("records", "items", "data", "results"):
            value = payload.get(key)
            if isinstance(value, list):
                return [item for item in value if isinstance(item, dict)]
    raise SystemExit(f"无法从 JSON 解析记录数组: {path}")


def infer_asin_from_records(records: list[dict[str, Any]], input_path: Path) -> str:
    for record in records:
        for key in ("asin", "ASIN"):
            value = normalize_space(record.get(key))
            if re.fullmatch(r"[A-Z0-9]{10}", value):
                return value
        review_link = normalize_space(
            record.get("评论链接网址") or record.get("review_link")
        )
        if review_link:
            try:
                asin, _ = parse_product_url(review_link)
                return asin
            except Exception:  # noqa: BLE001
                pass
    match = re.search(r"([A-Z0-9]{10})", input_path.stem, re.I)
    if match:
        return match.group(1).upper()
    return "amazon_reviews"


def default_build_stem(input_path: Path) -> str:
    stem = input_path.stem
    if stem.endswith("_review_rows_machine"):
        return stem.removesuffix("_review_rows_machine") + "_review_workbook"
    if stem.endswith("_review_rows_translated_labeled"):
        return stem.removesuffix("_review_rows_translated_labeled") + "_review_workbook"
    if stem.endswith("_review_rows_translated"):
        return stem.removesuffix("_review_rows_translated") + "_review_workbook"
    if stem.endswith("_review_rows_factual"):
        return stem.removesuffix("_review_rows_factual") + "_review_workbook"
    if stem.endswith("_translated"):
        return stem.removesuffix("_translated") + "_review_workbook"
    if stem.endswith("_labeled"):
        return stem
    return stem + "_review_workbook"


def build_factual_records(payload: dict[str, Any]) -> list[dict[str, Any]]:
    host = str(payload["host"])
    raw_rows = [item for item in payload.get("results", []) if isinstance(item, dict)]
    rows: list[dict[str, Any]] = []
    for index, item in enumerate(raw_rows, start=1):
        clean_title = strip_rating_prefix(str(item.get("title", "")))
        review_text = normalize_space(str(item.get("body", ""))) or clean_title
        rows.append(
            {
                "序号": str(index),
                "评论用户名": normalize_space(str(item.get("author", ""))),
                "国家": infer_country(str(item.get("country_date", "")), host),
                "星级评分": parse_rating(str(item.get("rating_text", ""))),
                "评论原文": review_text,
                "评论中文版": "",
                "评论概括": "",
                "情感倾向": "",
                "类别分类": "",
                "标签": "",
                "重点标记": "",
                "评论链接网址": build_review_link(
                    str(item.get("review_id", "")), host, item.get("review_link")
                ),
                "评论时间": normalize_review_time(
                    item.get("review_time") or item.get("country_date", "")
                ),
                "评论点赞数": normalize_helpful_votes(item.get("helpful_votes", "")),
                "host": host,
                "review_id": str(item.get("review_id", "")),
            }
        )
    return rows


def translate_records(
    records: list[dict[str, Any]],
    *,
    api_url: str,
    api_key: str,
    source_lang: str,
    target_lang: str,
    timeout: int,
    retries: int,
) -> list[dict[str, Any]]:
    texts_to_translate = []
    text_indices = []
    for index, record in enumerate(records):
        source_text = normalize_space(record.get("评论原文"))
        target_text = normalize_space(record.get("评论中文版"))
        if source_text and not target_text:
            texts_to_translate.append(source_text)
            text_indices.append(index)

    if texts_to_translate:
        translated = translate_texts_batch(
            texts_to_translate,
            api_url=api_url,
            [REDACTED_SECRET],
            source_lang=source_lang,
            target_lang=target_lang,
            timeout=timeout,
            retries=retries,
            batch_size=50,
        )
        for i, translated_text in enumerate(translated):
            record_index = text_indices[i]
            records[record_index]["评论中文版"] = translated_text
            if (i + 1) % 20 == 0:
                print(f"translate_progress={i + 1}/{len(texts_to_translate)}")

    return records


def command_doctor(args: argparse.Namespace) -> int:
    asin, host = parse_product_url(args.url)
    api_url = args.api_url or read_env_value("DEEPLX_API_URL") or ""
    [REDACTED_SECRET] or read_env_value("DEEPLX_API_KEY") or ""
    deeplx_ready = False
    deeplx_probe = "missing_api_url"
    if api_url:
        deeplx_ready, deeplx_probe = probe_deeplx(api_url, api_key)
    print(
        json.dumps(
            {"asin": asin, "host": host, "review_url": build_review_url(host, asin)},
            ensure_ascii=False,
        )
    )
    print(
        json.dumps(
            {
                "chrome_debug_ready": port_is_open("127.0.0.1", args.cdp_port),
                "cdp_port": args.cdp_port,
            },
            ensure_ascii=False,
        )
    )
    print(
        json.dumps(
            {
                "deeplx_env_ready": bool(api_url),
                "deeplx_api_url_present": bool(api_url),
                "deeplx_reachable": deeplx_ready,
                "deeplx_probe": deeplx_probe,
                "translation_fallback": "model" if not deeplx_ready else "deeplx",
            },
            ensure_ascii=False,
        )
    )
    return 0


def command_coverage_check(args: argparse.Namespace) -> int:
    asin, host = parse_product_url(args.url)
    review_url = build_review_url(host, asin)
    db_path = Path(args.db_path).expanduser()
    cached_count = 0
    searched_keyword_count = 0
    successful_keywords: list[dict[str, Any]] = []
    recent_zero_keywords: list[dict[str, Any]] = []
    if db_path.exists():
        conn = ensure_db(db_path)
        cached_count = get_cached_review_count(conn, host, asin)
        keyword_history = get_keyword_history_map(conn, host, asin)
        searched_keyword_count = len(keyword_history)
        for keyword, entry in keyword_history.items():
            if int(entry.get("best_new_count") or 0) > 0:
                successful_keywords.append(
                    {
                        "keyword": keyword,
                        "best_new_count": int(entry.get("best_new_count") or 0),
                        "search_count": int(entry.get("search_count") or 0),
                    }
                )
            else:
                recent_zero_keywords.append(
                    {
                        "keyword": keyword,
                        "searched_at": entry.get("searched_at", ""),
                    }
                )
        conn.close()
    successful_keywords = sorted(
        successful_keywords,
        key=lambda item: (-item["best_new_count"], item["keyword"]),
    )

    input_count = 0
    if args.input_json:
        input_count = len(load_layered_records(Path(args.input_json).expanduser()))

    chrome_debug_ready = port_is_open("127.0.0.1", args.cdp_port)
    page_probe: dict[str, Any] = {
        "page_total_reviews": None,
        "page_total_reviews_evidence": "",
        "page_total_ratings": None,
        "page_total_ratings_evidence": "",
        "probe_url": review_url,
        "probe_title": "",
        "probe_texts": [],
    }
    if chrome_debug_ready:
        try:
            page_probe = fetch_review_page_totals(review_url, args.cdp_port)
        except Exception as exc:  # noqa: BLE001
            page_probe["probe_error"] = str(exc)

    current_count = input_count or cached_count
    coverage_ratio = None
    if page_probe.get("page_total_reviews"):
        coverage_ratio = round(current_count / page_probe["page_total_reviews"], 3)

    print(
        json.dumps(
            {
                "asin": asin,
                "host": host,
                "review_url": review_url,
                "chrome_debug_ready": chrome_debug_ready,
                "current_count": current_count,
                "current_count_source": "input_json" if input_count else "cache_db",
                "cache_db_count": cached_count,
                "input_json_count": input_count,
                "searched_keyword_count": searched_keyword_count,
                "successful_keyword_count": len(successful_keywords),
                "successful_keywords_top": successful_keywords[:10],
                "recent_zero_keywords_top": recent_zero_keywords[:10],
                "coverage_ratio": coverage_ratio,
                **page_probe,
            },
            ensure_ascii=False,
            indent=2,
        )
    )
    return 0


def command_keyword_autotune(args: argparse.Namespace) -> int:
    output_dir = Path(args.output_dir).expanduser()
    db_path = Path(args.db_path).expanduser()
    output_path = (
        Path(args.keyword_tuning_state).expanduser()
        if args.keyword_tuning_state
        else default_keyword_tuning_state_path(output_dir)
    )
    report_paths: list[Path] = []
    for pattern in args.report_glob or []:
        for matched in glob.glob(pattern):
            report_paths.append(Path(matched).expanduser())
    payload = refresh_keyword_tuning_state(
        db_path=db_path,
        output_path=output_path,
        report_paths=report_paths,
        top_k=args.top_k,
    )
    summary = {
        "keyword_count": payload["metadata"]["keyword_count"],
        "top_k": payload["metadata"]["top_k"],
        "profiles": {
            profile: profile_payload.get("recommended_keywords", [])
            for profile, profile_payload in (payload.get("profiles") or {}).items()
        },
        "output_path": str(output_path),
    }
    print(json.dumps(summary, ensure_ascii=False, indent=2))
    return 0


def command_collect(args: argparse.Namespace) -> int:
    output_dir = Path(args.output_dir)
    payload = collect_reviews(
        args.url,
        port=args.cdp_port,
        max_pages=args.max_pages,
        resume_from=args.resume_from_page,
        mode=getattr(args, "mode", "deep"),
        db_path=Path(args.db_path)
        if hasattr(args, "db_path") and args.db_path
        else None,
        refresh_cache=getattr(args, "refresh_cache", False),
        keywords=getattr(args, "keywords", None),
        keyword_profile=getattr(args, "keyword_profile", DEFAULT_KEYWORD_PROFILE),
        keyword_reuse_scope=getattr(
            args, "keyword_reuse_scope", DEFAULT_KEYWORD_REUSE_SCOPE
        ),
        zero_result_retry_hours=getattr(
            args, "zero_result_retry_hours", DEFAULT_ZERO_RESULT_RETRY_HOURS
        ),
        keyword_tuning_state_path=Path(args.keyword_tuning_state).expanduser()
        if getattr(args, "keyword_tuning_state", "")
        else default_keyword_tuning_state_path(output_dir),
        combo_delay_seconds=getattr(args, "combo_delay", DEFAULT_COMBO_DELAY_SECONDS),
    )
    raw_json = build_collect_output_path(output_dir, payload["asin"])
    raw_json.parent.mkdir(parents=True, exist_ok=True)
    raw_json.write_text(
        json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8"
    )
    print(raw_json)
    print(
        json.dumps(
            {
                "asin": payload["asin"],
                "combo_count": payload.get("combo_count", 1),
                "keyword_count": payload.get("keyword_count", 0),
                "keyword_mode": payload.get("keyword_mode", "off"),
                "keyword_profile": payload.get("keyword_profile", ""),
                "skipped_keywords": payload.get("skipped_keywords", 0),
                "row_count": payload["row_count"],
            },
            ensure_ascii=False,
        )
    )
    return 0


def command_intake(args: argparse.Namespace) -> int:
    output_dir = Path(args.output_dir)
    payload = collect_reviews(
        args.url,
        port=args.cdp_port,
        max_pages=args.max_pages,
        resume_from=args.resume_from_page,
        mode=getattr(args, "mode", "deep"),
        db_path=Path(args.db_path)
        if hasattr(args, "db_path") and args.db_path
        else None,
        refresh_cache=getattr(args, "refresh_cache", False),
        keywords=getattr(args, "keywords", None),
        keyword_profile=getattr(args, "keyword_profile", DEFAULT_KEYWORD_PROFILE),
        keyword_reuse_scope=getattr(
            args, "keyword_reuse_scope", DEFAULT_KEYWORD_REUSE_SCOPE
        ),
        zero_result_retry_hours=getattr(
            args, "zero_result_retry_hours", DEFAULT_ZERO_RESULT_RETRY_HOURS
        ),
        keyword_tuning_state_path=Path(args.keyword_tuning_state).expanduser()
        if getattr(args, "keyword_tuning_state", "")
        else default_keyword_tuning_state_path(output_dir),
        combo_delay_seconds=getattr(args, "combo_delay", DEFAULT_COMBO_DELAY_SECONDS),
    )
    raw_json = build_collect_output_path(output_dir, payload["asin"])
    factual_json, factual_xlsx, factual_csv = build_factual_output_paths(
        output_dir, payload["asin"]
    )
    raw_json.parent.mkdir(parents=True, exist_ok=True)
    raw_json.write_text(
        json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8"
    )

    factual_records = build_factual_records(payload)
    result = write_delivery_artifacts(
        factual_records, factual_json, factual_xlsx, factual_csv, strict=False
    )
    if result["warnings"]:
        print("warnings=")
        for warning in result["warnings"]:
            print(f"- {warning}")

    print(raw_json)
    print(factual_json)
    print(factual_xlsx)
    print(factual_csv)
    return 0


def command_translate(args: argparse.Namespace) -> int:
    input_json = Path(args.input_json).expanduser()
    records = load_layered_records(input_json)
    asin = infer_asin_from_records(records, input_json)
    output_dir = Path(args.output_dir)
    output_json, output_xlsx, output_csv = build_translated_output_paths(
        output_dir, asin
    )
    translation_mode = "deeplx"
    translated_records = records
    deeplx_failed = False
    try:
        api_url = resolve_api_url(args.api_url)
        [REDACTED_SECRET])
        deeplx_ready, deeplx_probe = probe_deeplx(
            api_url, api_key, timeout=min(args.timeout, 12)
        )
        if not deeplx_ready:
            translation_mode = "model_fallback"
            deeplx_failed = True
            print(
                json.dumps(
                    {"translation_mode": translation_mode, "reason": deeplx_probe},
                    ensure_ascii=False,
                )
            )
        else:
            translated_records = translate_records(
                records,
                api_url=api_url,
                [REDACTED_SECRET],
                source_lang=args.source_lang,
                target_lang=args.target_lang,
                timeout=args.timeout,
                retries=args.retries,
            )
    except SystemExit as exc:
        translation_mode = "model_fallback"
        deeplx_failed = True
        reason = str(exc) if exc else "missing_deeplx_config"
        print(
            json.dumps(
                {"translation_mode": translation_mode, "reason": reason},
                ensure_ascii=False,
            )
        )

    if deeplx_failed:
        # Output pending translation list for AI to translate
        pending_texts = []
        for index, record in enumerate(translated_records):
            source = normalize_space(record.get("评论原文"))
            target = normalize_space(record.get("评论中文版"))
            if source and not target:
                pending_texts.append({"index": index + 1, "text": source})

        if pending_texts:
            pending_file = output_dir / f"amazon_{asin}_pending_translation.json"
            pending_file.parent.mkdir(parents=True, exist_ok=True)
            pending_file.write_text(
                json.dumps(pending_texts, ensure_ascii=False, indent=2),
                encoding="utf-8",
            )
            print(f"\nDeepLX 不可用，{len(pending_texts)} 条评论待翻译。")
            print(f"待翻译清单已保存：{pending_file}")
            print("请把待翻译文本发给 AI（Sen），由 AI 翻译后填回。")
            for record in translated_records:
                source = normalize_space(record.get("评论原文", ""))
                target = normalize_space(record.get("评论中文版", ""))
                if source and not target:
                    record["评论中文版"] = "[待翻译]"

    result = write_delivery_artifacts(
        translated_records, output_json, output_xlsx, output_csv, strict=False
    )
    if result["warnings"]:
        print("warnings=")
        for warning in result["warnings"]:
            print(f"- {warning}")
    print(json.dumps({"translation_mode": translation_mode}, ensure_ascii=False))
    print(output_json)
    print(output_xlsx)
    print(output_csv)
    return 0


def command_build(args: argparse.Namespace) -> int:
    input_json = Path(args.input_json).expanduser()
    records = load_json_records(input_json)
    output_dir = Path(args.output_dir)
    stem = args.output_stem or default_build_stem(input_json)
    output_json, output_xlsx, output_csv = build_final_output_paths(output_dir, stem)
    result = write_delivery_artifacts(
        records, output_json, output_xlsx, output_csv, strict=args.strict
    )
    if result["warnings"]:
        print("warnings=")
        for warning in result["warnings"]:
            print(f"- {warning}")
    print(output_json)
    print(output_xlsx)
    print(output_csv)
    return 0


def command_prepare_tagging(args: argparse.Namespace) -> int:
    input_json = Path(args.input_json).expanduser()
    records = load_layered_records(input_json)
    asin = infer_asin_from_records(records, input_json)
    output_dir = Path(args.output_dir)
    output_path = build_prepare_output_path(output_dir, asin)
    canonical_tags = (
        load_canonical_tags(Path(args.canonical_tags_json).expanduser())
        if args.canonical_tags_json
        else []
    )
    cache_path = (
        Path(args.cache_file).expanduser()
        if args.cache_file
        else default_cache_path(output_dir)
    )
    cache_entries = load_cache(cache_path, ttl_hours=args.cache_ttl_hours)
    payload = prepare_tagging_payload(
        records,
        taxonomy_version=args.taxonomy_version,
        cache_entries=cache_entries,
        canonical_tags=canonical_tags,
    )
    output_path.parent.mkdir(parents=True, exist_ok=True)
    output_path.write_text(
        json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8"
    )
    print(output_path)
    print(json.dumps(payload["metadata"], ensure_ascii=False))
    return 0


def command_taxonomy_bootstrap(args: argparse.Namespace) -> int:
    input_json = Path(args.input_json).expanduser()
    records = load_layered_records(input_json)
    asin = infer_asin_from_records(records, input_json)
    output_dir = Path(args.output_dir)
    output_path = build_bootstrap_output_path(output_dir, asin)
    payload = build_taxonomy_bootstrap(
        records,
        taxonomy_version=args.taxonomy_version,
        sample_size=args.sample_size,
    )
    output_path.parent.mkdir(parents=True, exist_ok=True)
    output_path.write_text(
        json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8"
    )
    print(output_path)
    print(json.dumps(payload["metadata"], ensure_ascii=False))
    return 0


def command_merge_build(args: argparse.Namespace) -> int:
    base_json = Path(args.base_json).expanduser()
    labels_json = Path(args.labels_json).expanduser()
    base_records = load_layered_records(base_json)
    labels_payload = json.loads(labels_json.read_text(encoding="utf-8"))
    output_dir = Path(args.output_dir)
    cache_path = (
        Path(args.cache_file).expanduser()
        if args.cache_file
        else default_cache_path(output_dir)
    )
    cache_entries = load_cache(cache_path, ttl_hours=args.cache_ttl_hours)
    merged_records, updated_cache, stats = merge_records_with_labels(
        base_records,
        labels_payload=labels_payload,
        taxonomy_version=args.taxonomy_version,
        cache_entries=cache_entries,
    )
    save_cache(cache_path, updated_cache)
    stem = args.output_stem or default_build_stem(base_json)
    output_json, output_xlsx, output_csv = build_final_output_paths(output_dir, stem)
    result = write_delivery_artifacts(
        merged_records, output_json, output_xlsx, output_csv, strict=args.strict
    )
    if result["warnings"]:
        print("warnings=")
        for warning in result["warnings"]:
            print(f"- {warning}")
    print(json.dumps({"cache_file": str(cache_path), **stats}, ensure_ascii=False))
    print(output_json)
    print(output_xlsx)
    print(output_csv)
    return 0


def print_summary(
    records: list[dict[str, Any]],
    *,
    asin: str = "",
    translation_ok: int = 0,
    cache_hits: int = 0,
    cache_total: int = 0,
) -> None:
    total = len(records)
    rating_counter: dict[str, int] = {}
    sentiment_counter: dict[str, int] = {}
    category_counter: dict[str, int] = {}
    for record in records:
        rating = normalize_space(record.get("星级评分"))
        if rating:
            rating_counter[rating] = rating_counter.get(rating, 0) + 1
        sentiment = normalize_space(record.get("情感倾向"))
        if sentiment:
            sentiment_counter[sentiment] = sentiment_counter.get(sentiment, 0) + 1
        category = normalize_space(record.get("类别分类"))
        if category:
            for cat in category.split(" / "):
                cat = cat.strip()
                if cat:
                    category_counter[cat] = category_counter.get(cat, 0) + 1

    summary = {
        "asin": asin,
        "total_reviews": total,
        "rating_distribution": dict(
            sorted(rating_counter.items(), key=lambda x: x[0], reverse=True)
        ),
        "sentiment_distribution": sentiment_counter,
        "category_distribution": dict(
            sorted(category_counter.items(), key=lambda x: x[1], reverse=True)
        ),
        "translation_filled": translation_ok,
        "cache_stats": {
            "hits": cache_hits,
            "total": cache_total,
            "hit_rate": round(cache_hits / cache_total, 2) if cache_total else 0,
        },
    }
    print(json.dumps({"summary": summary}, ensure_ascii=False, indent=2))


def command_batch_intake(args: argparse.Namespace) -> int:
    url_file = Path(args.url_file).expanduser()
    if not url_file.exists():
        raise SystemExit(f"URL 文件不存在: {url_file}")

    urls = [
        line.strip()
        for line in url_file.read_text(encoding="utf-8").splitlines()
        if line.strip() and not line.startswith("#")
    ]
    if not urls:
        raise SystemExit("URL 文件为空")

    output_dir = Path(args.output_dir)
    results: list[dict[str, Any]] = []
    for idx, url in enumerate(urls, start=1):
        print(f"[{idx}/{len(urls)}] processing: {url}")
        try:
            payload = collect_reviews(
                url,
                port=args.cdp_port,
                max_pages=args.max_pages,
                mode=getattr(args, "mode", "deep"),
                db_path=Path(args.db_path)
                if hasattr(args, "db_path") and args.db_path
                else None,
                refresh_cache=getattr(args, "refresh_cache", False),
                keywords=getattr(args, "keywords", None),
                keyword_profile=getattr(
                    args, "keyword_profile", DEFAULT_KEYWORD_PROFILE
                ),
                keyword_reuse_scope=getattr(
                    args, "keyword_reuse_scope", DEFAULT_KEYWORD_REUSE_SCOPE
                ),
                zero_result_retry_hours=getattr(
                    args,
                    "zero_result_retry_hours",
                    DEFAULT_ZERO_RESULT_RETRY_HOURS,
                ),
                keyword_tuning_state_path=Path(args.keyword_tuning_state).expanduser()
                if getattr(args, "keyword_tuning_state", "")
                else default_keyword_tuning_state_path(output_dir),
                combo_delay_seconds=getattr(
                    args, "combo_delay", DEFAULT_COMBO_DELAY_SECONDS
                ),
            )
            raw_json = build_collect_output_path(output_dir, payload["asin"])
            factual_json, factual_xlsx, factual_csv = build_factual_output_paths(
                output_dir, payload["asin"]
            )
            raw_json.parent.mkdir(parents=True, exist_ok=True)
            raw_json.write_text(
                json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8"
            )
            factual_records = build_factual_records(payload)
            result = write_delivery_artifacts(
                factual_records, factual_json, factual_xlsx, factual_csv, strict=False
            )
            results.append(
                {
                    "url": url,
                    "asin": payload["asin"],
                    "rows": payload["row_count"],
                    "status": "ok",
                }
            )
        except Exception as exc:  # noqa: BLE001
            results.append(
                {
                    "url": url,
                    "asin": "",
                    "rows": 0,
                    "status": "error",
                    "error": str(exc),
                }
            )

    summary_path = output_dir / "batch_intake_summary.json"
    summary_path.parent.mkdir(parents=True, exist_ok=True)
    summary_path.write_text(
        json.dumps(
            {"total_urls": len(urls), "results": results}, ensure_ascii=False, indent=2
        ),
        encoding="utf-8",
    )
    print(
        json.dumps(
            {
                "batch_summary": {
                    "total_urls": len(urls),
                    "success": sum(1 for r in results if r["status"] == "ok"),
                    "failed": sum(1 for r in results if r["status"] == "error"),
                }
            },
            ensure_ascii=False,
        )
    )
    print(summary_path)
    return 0


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        description="Collect Amazon reviews via Chrome CDP and export a 14-column workbook."
    )
    subparsers = parser.add_subparsers(dest="command", required=True)

    doctor = subparsers.add_parser(
        "doctor", help="Check URL parsing and Chrome debug availability."
    )
    doctor.add_argument("--url", required=True)
    doctor.add_argument("--cdp-port", type=int, default=DEFAULT_CDP_PORT)
    doctor.add_argument("--api-url", help="Optional DeepLX API URL override.")
    doctor.add_argument("--api-key", help="Optional DeepLX API key override.")

    coverage = subparsers.add_parser(
        "coverage-check",
        help="Compare current cached/input rows against Amazon's visible review totals.",
    )
    coverage.add_argument("--url", required=True)
    coverage.add_argument("--db-path", default=str(DEFAULT_DB_PATH))
    coverage.add_argument("--input-json")
    coverage.add_argument("--cdp-port", type=int, default=DEFAULT_CDP_PORT)

    autotune = subparsers.add_parser(
        "keyword-autotune",
        help="Aggregate historical keyword hit data and build a tuned keyword state file.",
    )
    autotune.add_argument("--output-dir", default=str(DEFAULT_OUTPUT_DIR))
    autotune.add_argument("--db-path", default=str(DEFAULT_DB_PATH))
    autotune.add_argument(
        "--report-glob",
        nargs="*",
        default=[],
        help="Optional glob patterns for historical keyword result JSON files.",
    )
    autotune.add_argument(
        "--keyword-tuning-state",
        help="Optional output path for the tuned keyword state JSON.",
    )
    autotune.add_argument(
        "--top-k",
        type=int,
        default=DEFAULT_KEYWORD_TUNING_TOP_K,
        help="How many recommended keywords to keep per profile. Default: 12",
    )

    collect = subparsers.add_parser(
        "collect", help="Collect all paginated reviews into raw JSON."
    )
    collect.add_argument("--url", required=True)
    collect.add_argument("--output-dir", default=str(DEFAULT_OUTPUT_DIR))
    collect.add_argument("--max-pages", type=int, default=DEFAULT_MAX_PAGES)
    collect.add_argument("--cdp-port", type=int, default=DEFAULT_CDP_PORT)
    collect.add_argument(
        "--resume-from-page",
        type=int,
        default=0,
        help="Skip pages already collected (1-based).",
    )
    collect.add_argument(
        "--mode",
        choices=["fast", "deep"],
        default="deep",
        help="fast=2 combos (~200 reviews), deep=18 combos only. Keywords are manual via --keywords.",
    )
    collect.add_argument(
        "--refresh-cache",
        action="store_true",
        help="Force re-collection even if cache exists.",
    )
    collect.add_argument(
        "--db-path",
        default=str(DEFAULT_DB_PATH),
        help="Path to SQLite cache file.",
    )
    collect.add_argument(
        "--keywords",
        nargs="*",
        help="Manual keyword pass. Omit the flag to skip keywords; pass --keywords with no values to use the default high-yield preset; or provide an explicit list.",
    )
    collect.add_argument(
        "--keyword-profile",
        choices=sorted(KEYWORD_PROFILES.keys()),
        default=DEFAULT_KEYWORD_PROFILE,
        help="Built-in keyword preset profile used when --keywords has no explicit values.",
    )
    collect.add_argument(
        "--keyword-reuse-scope",
        choices=["successful", "all", "none"],
        default=DEFAULT_KEYWORD_REUSE_SCOPE,
        help="successful=skip keywords that have produced results before; all=skip every previously searched keyword; none=always rerun keywords.",
    )
    collect.add_argument(
        "--zero-result-retry-hours",
        type=float,
        default=DEFAULT_ZERO_RESULT_RETRY_HOURS,
        help="How long to suppress recently searched zero-result keywords. Default: 72",
    )
    collect.add_argument(
        "--keyword-tuning-state",
        help="Optional tuned keyword state JSON. Defaults to <output-dir>/keyword_tuning_state.json if present.",
    )
    collect.add_argument(
        "--combo-delay",
        type=float,
        default=DEFAULT_COMBO_DELAY_SECONDS,
        help="Delay in seconds between combo or keyword requests. Default: 2.5",
    )

    intake = subparsers.add_parser(
        "intake",
        help="Collect reviews and export a factual workbook with analysis columns left blank.",
    )
    intake.add_argument("--url", required=True)
    intake.add_argument("--output-dir", default=str(DEFAULT_OUTPUT_DIR))
    intake.add_argument("--max-pages", type=int, default=DEFAULT_MAX_PAGES)
    intake.add_argument("--cdp-port", type=int, default=DEFAULT_CDP_PORT)
    intake.add_argument(
        "--resume-from-page",
        type=int,
        default=0,
        help="Skip pages already collected (1-based).",
    )
    intake.add_argument(
        "--mode",
        choices=["fast", "deep"],
        default="deep",
        help="fast=2 combos (~200 reviews), deep=18 combos only. Keywords are manual via --keywords.",
    )
    intake.add_argument(
        "--refresh-cache",
        action="store_true",
        help="Force re-collection even if cache exists.",
    )
    intake.add_argument(
        "--db-path",
        default=str(DEFAULT_DB_PATH),
        help="Path to SQLite cache file.",
    )
    intake.add_argument(
        "--keywords",
        nargs="*",
        help="Manual keyword pass. Omit the flag to skip keywords; pass --keywords with no values to use the default high-yield preset; or provide an explicit list.",
    )
    intake.add_argument(
        "--keyword-profile",
        choices=sorted(KEYWORD_PROFILES.keys()),
        default=DEFAULT_KEYWORD_PROFILE,
        help="Built-in keyword preset profile used when --keywords has no explicit values.",
    )
    intake.add_argument(
        "--keyword-reuse-scope",
        choices=["successful", "all", "none"],
        default=DEFAULT_KEYWORD_REUSE_SCOPE,
        help="successful=skip keywords that have produced results before; all=skip every previously searched keyword; none=always rerun keywords.",
    )
    intake.add_argument(
        "--zero-result-retry-hours",
        type=float,
        default=DEFAULT_ZERO_RESULT_RETRY_HOURS,
        help="How long to suppress recently searched zero-result keywords. Default: 72",
    )
    intake.add_argument(
        "--keyword-tuning-state",
        help="Optional tuned keyword state JSON. Defaults to <output-dir>/keyword_tuning_state.json if present.",
    )
    intake.add_argument(
        "--combo-delay",
        type=float,
        default=DEFAULT_COMBO_DELAY_SECONDS,
        help="Delay in seconds between combo or keyword requests. Default: 2.5",
    )

    translate = subparsers.add_parser(
        "translate",
        help="Fill 评论中文版 via DeepLX while preserving other factual columns.",
    )
    translate.add_argument("--input-json", required=True)
    translate.add_argument("--output-dir", default=str(DEFAULT_OUTPUT_DIR))
    translate.add_argument(
        "--api-url", help="DeepLX translate endpoint. Defaults to DEEPLX_API_URL env."
    )
    translate.add_argument(
        "--api-key", help="Optional DeepLX API key. Defaults to DEEPLX_API_KEY env."
    )
    translate.add_argument("--source-lang", default="auto")
    translate.add_argument("--target-lang", default="ZH")
    translate.add_argument("--timeout", type=int, default=30)
    translate.add_argument("--retries", type=int, default=3)

    build = subparsers.add_parser(
        "build",
        help="Build the final workbook from a JSON file that already contains labeled rows.",
    )
    build.add_argument("--input-json", required=True)
    build.add_argument("--output-dir", default=str(DEFAULT_OUTPUT_DIR))
    build.add_argument(
        "--output-stem", help="Optional custom file stem without extension."
    )
    build.add_argument(
        "--strict",
        action="store_true",
        help="Fail when required analysis fields are still missing.",
    )

    prepare = subparsers.add_parser(
        "prepare-tagging",
        help="Create a lightweight tagging JSON with cache-aware pending rows.",
    )
    prepare.add_argument("--input-json", required=True)
    prepare.add_argument("--output-dir", default=str(DEFAULT_OUTPUT_DIR))
    prepare.add_argument("--taxonomy-version", default="v1")
    prepare.add_argument("--cache-file", help="Optional label cache file path.")
    prepare.add_argument(
        "--cache-ttl-hours",
        type=float,
        default=0,
        help="Skip cache entries older than this many hours (0 = no expiry).",
    )
    prepare.add_argument(
        "--canonical-tags-json", help="Optional canonical tags JSON file."
    )

    bootstrap = subparsers.add_parser(
        "taxonomy-bootstrap",
        help="Create representative sample rows for canonical tag design.",
    )
    bootstrap.add_argument("--input-json", required=True)
    bootstrap.add_argument("--output-dir", default=str(DEFAULT_OUTPUT_DIR))
    bootstrap.add_argument("--taxonomy-version", default="v1")
    bootstrap.add_argument("--sample-size", type=int, default=30)

    merge_build = subparsers.add_parser(
        "merge-build",
        help="Merge factual/translated rows with labeled rows, update cache, and build final workbook.",
    )
    merge_build.add_argument("--base-json", required=True)
    merge_build.add_argument("--labels-json", required=True)
    merge_build.add_argument("--output-dir", default=str(DEFAULT_OUTPUT_DIR))
    merge_build.add_argument(
        "--output-stem", help="Optional custom file stem without extension."
    )
    merge_build.add_argument("--taxonomy-version", default="v1")
    merge_build.add_argument("--cache-file", help="Optional label cache file path.")
    merge_build.add_argument(
        "--cache-ttl-hours",
        type=float,
        default=0,
        help="Skip cache entries older than this many hours (0 = no expiry).",
    )
    merge_build.add_argument(
        "--strict",
        action="store_true",
        help="Fail when required analysis fields are still missing.",
    )

    batch = subparsers.add_parser(
        "batch-intake", help="Collect reviews from multiple URLs listed in a file."
    )
    batch.add_argument(
        "--url-file",
        required=True,
        help="Path to a text file with one Amazon URL per line.",
    )
    batch.add_argument("--output-dir", default=str(DEFAULT_OUTPUT_DIR))
    batch.add_argument("--max-pages", type=int, default=DEFAULT_MAX_PAGES)
    batch.add_argument("--cdp-port", type=int, default=DEFAULT_CDP_PORT)
    batch.add_argument(
        "--mode",
        choices=["fast", "deep"],
        default="deep",
        help="fast=2 combos (~200 reviews), deep=18 combos only. Keywords are manual via --keywords.",
    )
    batch.add_argument(
        "--db-path",
        default=str(DEFAULT_DB_PATH),
        help="Path to SQLite cache file.",
    )
    batch.add_argument(
        "--refresh-cache",
        action="store_true",
        help="Force re-collection even if cache exists.",
    )
    batch.add_argument(
        "--keywords",
        nargs="*",
        help="Manual keyword pass. Omit the flag to skip keywords; pass --keywords with no values to use the default high-yield preset; or provide an explicit list.",
    )
    batch.add_argument(
        "--keyword-profile",
        choices=sorted(KEYWORD_PROFILES.keys()),
        default=DEFAULT_KEYWORD_PROFILE,
        help="Built-in keyword preset profile used when --keywords has no explicit values.",
    )
    batch.add_argument(
        "--keyword-reuse-scope",
        choices=["successful", "all", "none"],
        default=DEFAULT_KEYWORD_REUSE_SCOPE,
        help="successful=skip keywords that have produced results before; all=skip every previously searched keyword; none=always rerun keywords.",
    )
    batch.add_argument(
        "--zero-result-retry-hours",
        type=float,
        default=DEFAULT_ZERO_RESULT_RETRY_HOURS,
        help="How long to suppress recently searched zero-result keywords. Default: 72",
    )
    batch.add_argument(
        "--keyword-tuning-state",
        help="Optional tuned keyword state JSON. Defaults to <output-dir>/keyword_tuning_state.json if present.",
    )
    batch.add_argument(
        "--combo-delay",
        type=float,
        default=DEFAULT_COMBO_DELAY_SECONDS,
        help="Delay in seconds between combo or keyword requests. Default: 2.5",
    )

    summary = subparsers.add_parser(
        "summary", help="Print a structured summary from an existing workbook JSON."
    )
    summary.add_argument("--input-json", required=True)
    summary.add_argument(
        "--cache-file", help="Optional label cache file for hit-rate stats."
    )

    return parser


def main() -> int:
    configure_stdio()
    parser = build_parser()
    args = parser.parse_args()
    if args.command == "doctor":
        return command_doctor(args)
    if args.command == "coverage-check":
        return command_coverage_check(args)
    if args.command == "keyword-autotune":
        return command_keyword_autotune(args)
    if args.command == "collect":
        return command_collect(args)
    if args.command == "intake":
        return command_intake(args)
    if args.command == "translate":
        return command_translate(args)
    if args.command == "build":
        return command_build(args)
    if args.command == "prepare-tagging":
        return command_prepare_tagging(args)
    if args.command == "taxonomy-bootstrap":
        return command_taxonomy_bootstrap(args)
    if args.command == "merge-build":
        return command_merge_build(args)
    if args.command == "batch-intake":
        return command_batch_intake(args)
    if args.command == "summary":
        input_json = Path(args.input_json).expanduser()
        records = load_layered_records(input_json)
        asin = infer_asin_from_records(records, input_json)
        cache_hits = 0
        cache_total = 0
        if args.cache_file:
            cache_entries = load_cache(Path(args.cache_file).expanduser())
            cache_total = len(cache_entries)
            cache_hits = sum(1 for v in cache_entries.values() if v.get("fields"))
        print_summary(
            records, asin=asin, cache_hits=cache_hits, cache_total=cache_total
        )
        return 0
    raise SystemExit(f"Unsupported command: {args.command}")


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/deeplx_translate.py
content: from __future__ import annotations

import json
import os
import time
from pathlib import Path
from typing import Iterable

import requests

# Unicode Private Use Area character used as batch separator.
# DeepLX preserves this character through translation, allowing us to
# reliably split the combined result back into individual translations.
_BATCH_SEP = "\ue001"


def parse_translation(payload: object) -> str:
    if isinstance(payload, str):
        return payload.strip()
    if isinstance(payload, dict):
        for key in ("data", "translation", "translatedText", "result", "text"):
            value = payload.get(key)
            if isinstance(value, str) and value.strip():
                return value.strip()
        alternatives = payload.get("alternatives")
        if isinstance(alternatives, list) and alternatives:
            first = alternatives[0]
            if isinstance(first, str) and first.strip():
                return first.strip()
    raise ValueError(f"Unsupported DeepLX response payload: {payload!r}")


def candidate_env_files() -> list[Path]:
    files = [
        Path.cwd() / ".env",
        Path(__file__).resolve().parents[1] / ".env",
    ]
    deduped: list[Path] = []
    seen: set[Path] = set()
    for path in files:
        if path not in seen:
            seen.add(path)
            deduped.append(path)
    return deduped


def read_env_value(key: str) -> str:
    explicit = os.getenv(key)
    if explicit:
        return explicit
    for env_path in candidate_env_files():
        if not env_path.exists():
            continue
        for line in env_path.read_text(encoding="utf-8").splitlines():
            stripped = line.strip()
            if not stripped or stripped.startswith("#") or "=" not in stripped:
                continue
            env_key, env_value = stripped.split("=", 1)
            if env_key.strip() != key:
                continue
            value = env_value.strip().strip("'").strip('"')
            if value:
                return value
    return ""


def resolve_api_url(explicit: str | None = None) -> str:
    api_url = explicit or read_env_value("DEEPLX_API_URL") or ""
    if not api_url:
        raise SystemExit("缺少 DEEPLX_API_URL。")
    return api_url


def resolve_api_key(explicit: str | None = None) -> str:
    return explicit or read_env_value("DEEPLX_API_KEY") or ""


def probe_deeplx(
    api_url: str, api_key: str = "", *, timeout: int = 12
) -> tuple[bool, str]:
    try:
        translated = translate_text(
            "hello",
            api_url=api_url,
            [REDACTED_SECRET],
            timeout=timeout,
            retries=1,
        )
        return True, translated
    except Exception as exc:  # noqa: BLE001
        return False, str(exc)


def call_deeplx(
    text: str,
    *,
    api_url: str,
    api_key: str = "",
    source_lang: str = "auto",
    target_lang: str = "ZH",
    timeout: int = 30,
) -> str:
    headers = {"Content-Type": "application/json"}
    if [REDACTED_SECRET]"Authorization"] = f"Bearer {api_key}"
        headers["x-api-key"] = api_key
        headers["apikey"] = api_key

    response = requests.post(
        api_url,
        headers=headers,
        data=json.dumps(
            {
                "text": text,
                "source_lang": source_lang,
                "target_lang": target_lang,
            }
        ),
        timeout=timeout,
    )
    response.raise_for_status()
    try:
        payload = response.json()
    except ValueError:
        payload = response.text
    return parse_translation(payload)


def translate_text(
    text: str,
    *,
    api_url: str,
    api_key: str = "",
    source_lang: str = "auto",
    target_lang: str = "ZH",
    timeout: int = 30,
    retries: int = 3,
    retry_sleep_seconds: float = 0.8,
) -> str:
    last_error: Exception | None = None
    for attempt in range(retries):
        try:
            return call_deeplx(
                text,
                api_url=api_url,
                [REDACTED_SECRET],
                source_lang=source_lang,
                target_lang=target_lang,
                timeout=timeout,
            )
        except Exception as exc:  # noqa: BLE001
            last_error = exc
            if attempt + 1 < retries:
                time.sleep(retry_sleep_seconds)
    raise RuntimeError(
        f"DeepLX translation failed after {retries} attempts: {last_error}"
    )


def _join_for_translation(texts: list[str]) -> str:
    """Join multiple texts with a Unicode PUA separator that DeepLX preserves."""
    return _BATCH_SEP.join(texts)


def _split_translated(
    combined: str,
    original_count: int,
) -> list[str]:
    """Split the combined translation back into individual results.

    Uses the PUA separator to split. If the separator count doesn't match,
    falls back to safe defaults to prevent index misalignment.
    """
    sep_count = combined.count(_BATCH_SEP)
    expected_sep = original_count - 1

    if sep_count == expected_sep:
        parts = combined.split(_BATCH_SEP)
        return [p.strip() for p in parts]

    # Fallback: separator count mismatch (shouldn't happen with PUA char,
    # but guard against it anyway)
    if sep_count > expected_sep:
        # Too many separators — take first N parts
        parts = combined.split(_BATCH_SEP)[:original_count]
        # Pad if still short
        while len(parts) < original_count:
            parts.append("")
        return [p.strip() for p in parts]

    # Too few or zero separators — return empty strings to signal failure
    return [""] * original_count


def call_deeplx_batch(
    texts: list[str],
    *,
    api_url: str,
    api_key: str = "",
    source_lang: str = "auto",
    target_lang: str = "ZH",
    timeout: int = 60,
) -> list[str]:
    headers = {"Content-Type": "application/json"}
    if [REDACTED_SECRET]"Authorization"] = f"Bearer {api_key}"
        headers["x-api-key"] = api_key
        headers["apikey"] = api_key

    combined = _join_for_translation(texts)

    response = requests.post(
        api_url,
        headers=headers,
        data=json.dumps(
            {
                "text": combined,
                "source_lang": source_lang,
                "target_lang": target_lang,
            }
        ),
        timeout=timeout,
    )
    response.raise_for_status()
    try:
        payload = response.json()
    except ValueError:
        raise ValueError(
            f"DeepLX batch response is not valid JSON: {response.text[:200]}"
        )

    translated = parse_translation(payload)
    return _split_translated(translated, len(texts))


def translate_texts_batch(
    texts: list[str],
    *,
    api_url: str,
    api_key: str = "",
    source_lang: str = "auto",
    target_lang: str = "ZH",
    timeout: int = 60,
    retries: int = 3,
    batch_size: int = 50,
    retry_sleep_seconds: float = 1.0,
) -> list[str]:
    all_results: list[str] = [""] * len(texts)
    batches = [texts[i : i + batch_size] for i in range(0, len(texts), batch_size)]

    for batch_idx, batch in enumerate(batches):
        last_error: Exception | None = None
        start_idx = batch_idx * batch_size
        batch_failed_completely = False
        for attempt in range(retries):
            try:
                results = call_deeplx_batch(
                    batch,
                    api_url=api_url,
                    [REDACTED_SECRET],
                    source_lang=source_lang,
                    target_lang=target_lang,
                    timeout=timeout,
                )
                for j, result in enumerate(results):
                    if start_idx + j < len(all_results):
                        all_results[start_idx + j] = result
                break
            except Exception as exc:  # noqa: BLE001
                last_error = exc
                error_str = str(exc).lower()
                if "400" in error_str and attempt == 0:
                    batch_failed_completely = True
                    break
                if attempt + 1 < retries:
                    time.sleep(retry_sleep_seconds)
        else:
            if not batch_failed_completely:
                for j in range(len(batch)):
                    if start_idx + j < len(all_results):
                        all_results[start_idx + j] = f"[翻译失败: {last_error}]"

        if batch_failed_completely:
            for j, text in enumerate(batch):
                idx = start_idx + j
                if idx >= len(all_results):
                    break
                try:
                    result = translate_text(
                        text,
                        api_url=api_url,
                        [REDACTED_SECRET],
                        source_lang=source_lang,
                        target_lang=target_lang,
                        timeout=min(timeout, 15),
                        retries=2,
                    )
                    all_results[idx] = result
                except Exception:  # noqa: BLE001
                    all_results[idx] = "[待翻译]"
                if (j + 1) % 10 == 0:
                    print(f"translate_fallback={idx + 1}/{len(texts)}")

    return all_results


def translate_texts(
    texts: Iterable[str],
    *,
    api_url: str,
    api_key: str = "",
    source_lang: str = "auto",
    target_lang: str = "ZH",
    timeout: int = 30,
    retries: int = 3,
    retry_sleep_seconds: float = 0.8,
) -> list[str]:
    return [
        translate_text(
            text,
            api_url=api_url,
            [REDACTED_SECRET],
            source_lang=source_lang,
            target_lang=target_lang,
            timeout=timeout,
            retries=retries,
            retry_sleep_seconds=retry_sleep_seconds,
        )
        for text in texts
    ]


path: scripts/label_workflow.py
content: from __future__ import annotations

import hashlib
import json
import re
import time
from collections import Counter, defaultdict
from pathlib import Path
from typing import Any

from review_delivery_schema import (
    normalize_categories,
    normalize_focus_marks,
    normalize_sentiment,
    normalize_space,
    normalize_tags,
)


def dedupe_preserve_order(items: list[str]) -> list[str]:
    seen: set[str] = set()
    result: list[str] = []
    for item in items:
        if item not in seen:
            seen.add(item)
            result.append(item)
    return result


ALLOWED_CATEGORIES = [
    "Praise on product",
    "Questions/Worrying",
    "Negative emotions",
    "Suggestion",
    "Nothing particular",
    "Being supportive to the brand",
    "Expecting",
    "Being Sarcastic",
    "Competitor comparison",
    "Status update",
]

FOCUS_MARK_OPTIONS = [
    "性价比",
    "质量",
    "功能改进建议",
    "竞品/型号",
    "软件/设置",
    "安装/适配",
    "品牌/售后",
    "物流/包装",
    "配件/兼容性",
]

NESTED_RECORD_KEYS = (
    "machine",
    "delivery",
    "fields",
    "analysis",
    "semantic",
    "payload",
    "data",
    "row",
    "record",
)

LABEL_OUTPUT_KEYS = (
    "评论概括",
    "summary",
    "summary_cn",
    "情感倾向",
    "sentiment",
    "类别分类",
    "category",
    "categories",
    "标签",
    "tag",
    "tags",
    "tags_cn",
    "重点标记",
    "focus_mark",
    "focus_marks",
    "priority_mark",
)


def flatten_record(raw: Any) -> dict[str, Any]:
    if not isinstance(raw, dict):
        return {}

    flattened: dict[str, Any] = {}
    for nested_key in NESTED_RECORD_KEYS:
        nested = raw.get(nested_key)
        if isinstance(nested, dict):
            flattened.update(flatten_record(nested))

    for key, value in raw.items():
        if key in NESTED_RECORD_KEYS and isinstance(value, dict):
            continue
        flattened[key] = value

    return flattened


def first_nonempty(source: Any, *names: str) -> str:
    record = flatten_record(source)
    for name in names:
        value = normalize_space(record.get(name))
        if value:
            return value
    return ""


def has_semantic_output(source: Any) -> bool:
    record = flatten_record(source)
    return any(normalize_space(record.get(key)) for key in LABEL_OUTPUT_KEYS)


def normalize_text_for_cache(text: Any) -> str:
    value = normalize_space(text).lower()
    value = re.sub(r"\s+", " ", value)
    return value


def build_cache_key(review_id: Any, review_text: Any, taxonomy_version: str) -> str:
    raw = f"{normalize_space(review_id)}\n{normalize_text_for_cache(review_text)}\n{normalize_space(taxonomy_version)}"
    return hashlib.sha256(raw.encode("utf-8")).hexdigest()


def default_cache_path(output_dir: Path) -> Path:
    return output_dir.expanduser() / "label_cache.jsonl"


def load_cache(path: Path, *, ttl_hours: float = 0) -> dict[str, dict[str, Any]]:
    if not path.exists():
        return {}
    mapping: dict[str, dict[str, Any]] = {}
    cutoff = time.time() - (ttl_hours * 3600) if ttl_hours > 0 else 0
    for line in path.read_text(encoding="utf-8").splitlines():
        line = line.strip()
        if not line:
            continue
        try:
            item = json.loads(line)
        except json.JSONDecodeError:
            continue
        if not isinstance(item, dict) or not item.get("cache_key"):
            continue
        if cutoff > 0:
            cached_at = item.get("cached_at", 0)
            if isinstance(cached_at, (int, float)) and cached_at < cutoff:
                continue
        mapping[str(item["cache_key"])] = item
    return mapping


def save_cache(path: Path, entries: dict[str, dict[str, Any]]) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    lines = [json.dumps(entries[key], ensure_ascii=False) for key in sorted(entries)]
    path.write_text("\n".join(lines) + ("\n" if lines else ""), encoding="utf-8")


def heuristic_sentiment(review_text: str, rating: str) -> tuple[str | None, bool]:
    text = normalize_text_for_cache(review_text)
    try:
        numeric_rating = float(rating) if rating else None
    except ValueError:
        numeric_rating = None

    positive_tokens = (
        "great",
        "excellent",
        "love",
        "perfect",
        "amazing",
        "good",
        "fantastic",
        "easy",
        "recommend",
    )
    negative_tokens = (
        "bad",
        "poor",
        "terrible",
        "worst",
        "awful",
        "broken",
        "refund",
        "problem",
        "issue",
        "disappointed",
    )
    suggestion_tokens = ("should", "wish", "suggest", "could", "希望", "建议", "改进")

    if numeric_rating is not None:
        if (
            numeric_rating >= 4.5
            and any(token in text for token in positive_tokens)
            and not any(token in text for token in negative_tokens)
        ):
            return "Positive", True
        if numeric_rating <= 2.5 and any(token in text for token in negative_tokens):
            return "Negative", True
        if numeric_rating == 3 and any(token in text for token in suggestion_tokens):
            return "Neutral", False
    return None, False


def heuristic_category(review_text: str, rating: str) -> tuple[str | None, bool]:
    text = normalize_text_for_cache(review_text)
    if "?" in review_text or any(
        token in text
        for token in ("anyone", "wonder", "question", "how do", "can i", "whether")
    ):
        return "Questions/Worrying", True
    if any(
        token in text for token in ("should", "wish", "suggest", "希望", "建议", "改进")
    ):
        return "Suggestion", True
    if any(
        token in text
        for token in ("competitor", "vs ", "compared", "better than", "比", "型号")
    ):
        return "Competitor comparison", True
    sentiment, confident = heuristic_sentiment(review_text, rating)
    if confident and sentiment == "Positive":
        return "Praise on product", True
    if confident and sentiment == "Negative":
        return "Negative emotions", True
    return None, False


def heuristic_focus_marks(
    review_text: str, translated_text: str = ""
) -> tuple[str, bool]:
    text = normalize_text_for_cache(" ".join([review_text, translated_text]))
    marks: list[str] = []
    if any(
        token in text for token in ("price", "value", "worth", "性价比", "贵", "便宜")
    ):
        marks.append("性价比")
    if any(token in text for token in ("quality", "durable", "build", "质量", "做工")):
        marks.append("质量")
    if any(
        token in text for token in ("suggest", "should", "wish", "希望", "建议", "改进")
    ):
        marks.append("功能改进建议")
    if any(token in text for token in ("compare", "vs", "competitor", "型号", "竞品")):
        marks.append("竞品/型号")
    if any(token in text for token in ("app", "software", "setup", "设置", "软件")):
        marks.append("软件/设置")
    if any(token in text for token in ("install", "fit", "mount", "安装", "适配")):
        marks.append("安装/适配")
    if any(
        token in text
        for token in ("support", "service", "brand", "售后", "客服", "品牌")
    ):
        marks.append("品牌/售后")
    if any(
        token in text for token in ("delivery", "package", "packaging", "物流", "包装")
    ):
        marks.append("物流/包装")
    if any(
        token in text
        for token in ("accessory", "compatible", "compatibility", "配件", "兼容")
    ):
        marks.append("配件/兼容性")
    return "；".join(dict.fromkeys(marks)), bool(marks)


def lightweight_row(record: dict[str, Any]) -> dict[str, Any]:
    record = flatten_record(record)
    review_text = first_nonempty(
        record,
        "评论原文",
        "review_text",
        "body",
        "content",
        "review_body",
        "title",
        "review_title",
    )
    translated = first_nonempty(
        record, "评论中文版", "translated_text", "translated_cn"
    )
    rating = first_nonempty(record, "星级评分", "rating", "star_rating", "rating_text")
    sentiment, sentiment_confident = heuristic_sentiment(review_text, rating)
    category, category_confident = heuristic_category(review_text, rating)
    focus_marks, focus_confident = heuristic_focus_marks(review_text, translated)
    return {
        "seq": first_nonempty(record, "序号", "seq", "index"),
        "review_id": first_nonempty(record, "review_id", "reviewId"),
        "review_text": review_text,
        "translated_text": translated,
        "rating": rating,
        "country": first_nonempty(
            record, "国家", "country", "country_cn", "region", "market"
        ),
        "prefilled_sentiment": sentiment or "",
        "prefilled_category": category or "",
        "prefilled_focus_marks": focus_marks,
        "heuristic_confidence": {
            "sentiment": sentiment_confident,
            "category": category_confident,
            "focus_marks": focus_confident,
        },
    }


def load_canonical_tags(path: Path | None) -> list[str]:
    if path is None or not path.exists():
        return []
    payload = json.loads(path.read_text(encoding="utf-8"))
    if isinstance(payload, list):
        return dedupe_preserve_order(
            [normalize_space(item) for item in payload if normalize_space(item)]
        )
    if isinstance(payload, dict):
        values = payload.get("canonical_tags") or payload.get("tags") or []
        if isinstance(values, list):
            return dedupe_preserve_order(
                [normalize_space(item) for item in values if normalize_space(item)]
            )
    return []


def prepare_tagging_payload(
    records: list[dict[str, Any]],
    *,
    taxonomy_version: str,
    cache_entries: dict[str, dict[str, Any]],
    canonical_tags: list[str],
) -> dict[str, Any]:
    items: list[dict[str, Any]] = []
    cached_rows: list[dict[str, Any]] = []
    for record in records:
        record = flatten_record(record)
        seq = first_nonempty(record, "序号", "seq", "index")
        review_id = first_nonempty(record, "review_id", "reviewId")
        review_text = first_nonempty(
            record,
            "评论原文",
            "review_text",
            "body",
            "content",
            "review_body",
            "title",
            "review_title",
        )
        cache_key = build_cache_key(review_id or seq, review_text, taxonomy_version)
        cached = cache_entries.get(cache_key)
        if cached:
            cached_rows.append(
                {
                    "seq": seq,
                    "review_id": review_id,
                    "cache_key": cache_key,
                }
            )
            continue

        item = lightweight_row(record)
        item["cache_key"] = cache_key
        item["allowed_categories"] = ALLOWED_CATEGORIES
        item["allowed_focus_marks"] = FOCUS_MARK_OPTIONS
        item["canonical_tags"] = canonical_tags
        items.append(item)

    return {
        "metadata": {
            "taxonomy_version": taxonomy_version,
            "total_rows": len(records),
            "pending_rows": len(items),
            "cached_rows": len(cached_rows),
        },
        "allowed_categories": ALLOWED_CATEGORIES,
        "allowed_focus_marks": FOCUS_MARK_OPTIONS,
        "canonical_tags": canonical_tags,
        "items": items,
        "cached_rows": cached_rows,
    }


def representative_sample(
    records: list[dict[str, Any]], sample_size: int
) -> list[dict[str, Any]]:
    records = [flatten_record(record) for record in records]
    buckets: dict[str, list[dict[str, Any]]] = defaultdict(list)
    for record in records:
        rating = (
            first_nonempty(record, "星级评分", "rating", "star_rating", "rating_text")
            or "unknown"
        )
        buckets[rating].append(record)

    sample: list[dict[str, Any]] = []
    for bucket in sorted(buckets):
        rows = sorted(
            buckets[bucket],
            key=lambda item: len(
                first_nonempty(
                    item,
                    "评论原文",
                    "review_text",
                    "body",
                    "content",
                    "review_body",
                    "title",
                    "review_title",
                )
            ),
            reverse=True,
        )
        if rows:
            sample.append(rows[0])

    if len(sample) >= sample_size:
        return sample[:sample_size]

    remaining = [row for row in records if row not in sample]
    remaining = sorted(
        remaining,
        key=lambda item: len(
            first_nonempty(
                item,
                "评论原文",
                "review_text",
                "body",
                "content",
                "review_body",
                "title",
                "review_title",
            )
        ),
        reverse=True,
    )
    for row in remaining:
        if len(sample) >= sample_size:
            break
        sample.append(row)
    return sample


def seed_tags(records: list[dict[str, Any]], limit: int = 20) -> list[str]:
    counter: Counter[str] = Counter()
    for record in records:
        record = flatten_record(record)
        text = normalize_text_for_cache(
            first_nonempty(
                record,
                "评论原文",
                "review_text",
                "body",
                "content",
                "review_body",
                "title",
                "review_title",
            )
        )
        for token in re.findall(r"[a-z]{4,}", text):
            if token in {
                "this",
                "that",
                "with",
                "have",
                "been",
                "very",
                "easy",
                "good",
            }:
                continue
            counter[token] += 1
    return [token for token, _ in counter.most_common(limit)]


def build_taxonomy_bootstrap(
    records: list[dict[str, Any]],
    *,
    taxonomy_version: str,
    sample_size: int,
) -> dict[str, Any]:
    records = [flatten_record(record) for record in records]
    sample = representative_sample(records, sample_size)
    sample_rows = []
    for record in sample:
        sample_rows.append(
            {
                "seq": first_nonempty(record, "序号", "seq", "index"),
                "review_id": first_nonempty(record, "review_id", "reviewId"),
                "review_text": first_nonempty(
                    record,
                    "评论原文",
                    "review_text",
                    "body",
                    "content",
                    "review_body",
                    "title",
                    "review_title",
                ),
                "translated_text": first_nonempty(
                    record, "评论中文版", "translated_text", "translated_cn"
                ),
                "rating": first_nonempty(
                    record, "星级评分", "rating", "star_rating", "rating_text"
                ),
                "country": first_nonempty(
                    record, "国家", "country", "country_cn", "region", "market"
                ),
            }
        )
    return {
        "metadata": {
            "taxonomy_version": taxonomy_version,
            "sample_size": len(sample_rows),
            "source_rows": len(records),
        },
        "allowed_categories": ALLOWED_CATEGORIES,
        "allowed_focus_marks": FOCUS_MARK_OPTIONS,
        "suggested_seed_tags": seed_tags(sample),
        "sample_rows": sample_rows,
        "instructions": {
            "goal": "Read sample_rows and produce a canonical tag vocabulary for this batch.",
            "output": "Write canonical_tags as a short stable Chinese tag list, then use it in prepare-tagging output.",
        },
    }


def parse_labels_payload(payload: Any) -> dict[str, dict[str, Any]]:
    if isinstance(payload, dict):
        items = payload.get("items") if isinstance(payload.get("items"), list) else None
        if items is None:
            items = (
                payload.get("labels")
                if isinstance(payload.get("labels"), list)
                else None
            )
        if items is None:
            items = (
                payload.get("rows") if isinstance(payload.get("rows"), list) else None
            )
        if items is None:
            items = (
                payload.get("records")
                if isinstance(payload.get("records"), list)
                else None
            )
        if items is None:
            items = [payload]
    elif isinstance(payload, list):
        items = payload
    else:
        items = []

    mapping: dict[str, dict[str, Any]] = {}
    for item in items:
        if not isinstance(item, dict):
            continue
        item = flatten_record(item)
        if not has_semantic_output(item):
            continue
        seq = first_nonempty(item, "seq", "序号", "index")
        review_id = first_nonempty(item, "review_id", "reviewId")
        key = review_id or seq
        if key:
            mapping[key] = item
    return mapping


def merged_semantic_fields(source: dict[str, Any]) -> dict[str, str]:
    source = flatten_record(source)
    return {
        "评论中文版": normalize_space(
            source.get("评论中文版")
            or source.get("translated_text")
            or source.get("translated_cn")
        ),
        "评论概括": normalize_space(
            source.get("评论概括") or source.get("summary") or source.get("summary_cn")
        ),
        "情感倾向": normalize_sentiment(
            source.get("情感倾向") or source.get("sentiment")
        ),
        "类别分类": normalize_categories(
            source.get("类别分类") or source.get("category") or source.get("categories")
        ),
        "标签": normalize_tags(
            source.get("标签")
            or source.get("tag")
            or source.get("tags")
            or source.get("tags_cn")
        ),
        "重点标记": normalize_focus_marks(
            source.get("重点标记")
            or source.get("focus_marks")
            or source.get("focus_mark")
        ),
    }


def combine_semantic_fields(
    base: dict[str, Any], override: dict[str, Any]
) -> dict[str, str]:
    base_fields = merged_semantic_fields(base)
    override_fields = merged_semantic_fields(override)
    combined: dict[str, str] = {}
    for key, base_value in base_fields.items():
        combined[key] = override_fields.get(key) or base_value
    return combined


def merge_records_with_labels(
    base_records: list[dict[str, Any]],
    *,
    labels_payload: Any,
    taxonomy_version: str,
    cache_entries: dict[str, dict[str, Any]],
) -> tuple[list[dict[str, Any]], dict[str, dict[str, Any]], dict[str, int]]:
    label_mapping = parse_labels_payload(labels_payload)
    merged: list[dict[str, Any]] = []
    updated_cache = dict(cache_entries)
    stats = {"cache_hits": 0, "label_hits": 0, "unresolved": 0}

    for record in base_records:
        row = flatten_record(record)
        seq = first_nonempty(row, "序号", "seq", "index")
        review_id = first_nonempty(row, "review_id", "reviewId")
        review_text = first_nonempty(
            row,
            "评论原文",
            "review_text",
            "body",
            "content",
            "review_body",
            "title",
            "review_title",
        )
        cache_key = build_cache_key(review_id or seq, review_text, taxonomy_version)
        source = None
        if review_id and review_id in label_mapping:
            source = label_mapping[review_id]
            stats["label_hits"] += 1
        elif seq and seq in label_mapping:
            source = label_mapping[seq]
            stats["label_hits"] += 1
        elif cache_key in cache_entries:
            source = cache_entries[cache_key].get("fields")
            stats["cache_hits"] += 1

        if source:
            combined_fields = combine_semantic_fields(row, source)
            row.update(combined_fields)
            updated_cache[cache_key] = {
                "cache_key": cache_key,
                "taxonomy_version": taxonomy_version,
                "review_id": review_id,
                "cached_at": time.time(),
                "fields": combined_fields,
            }
        else:
            stats["unresolved"] += 1
        merged.append(row)

    return merged, updated_cache, stats


path: scripts/review_cache.py
content: """SQLite cache for Amazon reviews — persistent storage to avoid re-fetching."""

from __future__ import annotations

import json
import sqlite3
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

DEFAULT_DB_PATH = Path.cwd() / "amazon-review-output" / "amazon_review_cache.sqlite3"


def utc_now() -> str:
    return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")


def ensure_db(db_path: Path = DEFAULT_DB_PATH) -> sqlite3.Connection:
    """Create tables if they don't exist and return connection."""
    db_path.parent.mkdir(parents=True, exist_ok=True)
    conn = sqlite3.connect(str(db_path))
    conn.execute("PRAGMA journal_mode=WAL")

    conn.execute("""
        CREATE TABLE IF NOT EXISTS jobs (
            job_id TEXT PRIMARY KEY,
            host TEXT NOT NULL,
            asin TEXT NOT NULL,
            source_url TEXT NOT NULL,
            started_at TEXT NOT NULL,
            finished_at TEXT,
            status TEXT NOT NULL DEFAULT 'running',
            current_seen_count INTEGER DEFAULT 0,
            new_review_count INTEGER DEFAULT 0,
            cached_total INTEGER DEFAULT 0,
            stats_json TEXT DEFAULT '[]'
        )
    """)

    conn.execute("""
        CREATE TABLE IF NOT EXISTS reviews (
            host TEXT NOT NULL,
            asin TEXT NOT NULL,
            review_id TEXT NOT NULL,
            title TEXT NOT NULL DEFAULT '',
            body TEXT NOT NULL DEFAULT '',
            rating_text TEXT NOT NULL DEFAULT '',
            country_date TEXT NOT NULL DEFAULT '',
            review_time TEXT NOT NULL DEFAULT '',
            helpful_votes TEXT NOT NULL DEFAULT '',
            author TEXT NOT NULL DEFAULT '',
            review_link TEXT NOT NULL DEFAULT '',
            source_combo TEXT NOT NULL DEFAULT '',
            first_seen_at TEXT NOT NULL,
            last_seen_at TEXT NOT NULL,
            last_job_id TEXT NOT NULL,
            PRIMARY KEY (host, asin, review_id)
        )
    """)

    conn.execute("""
        CREATE TABLE IF NOT EXISTS analyses (
            host TEXT NOT NULL,
            asin TEXT NOT NULL,
            review_id TEXT NOT NULL,
            country TEXT NOT NULL DEFAULT '',
            rating TEXT NOT NULL DEFAULT '',
            review_text TEXT NOT NULL DEFAULT '',
            translated_text TEXT NOT NULL DEFAULT '',
            summary TEXT NOT NULL DEFAULT '',
            sentiment TEXT NOT NULL DEFAULT '',
            categories TEXT NOT NULL DEFAULT '',
            tags TEXT NOT NULL DEFAULT '',
            focus_marks TEXT NOT NULL DEFAULT '',
            updated_at TEXT NOT NULL,
            PRIMARY KEY (host, asin, review_id)
        )
    """)

    conn.execute("""
        CREATE TABLE IF NOT EXISTS keyword_history (
            host TEXT NOT NULL,
            asin TEXT NOT NULL,
            keyword TEXT NOT NULL,
            searched_at TEXT NOT NULL,
            new_count INTEGER DEFAULT 0,
            best_new_count INTEGER DEFAULT 0,
            search_count INTEGER DEFAULT 1,
            total_at_search INTEGER DEFAULT 0,
            job_id TEXT NOT NULL,
            PRIMARY KEY (host, asin, keyword)
        )
    """)

    # Schema evolution: add missing columns
    _ensure_column(conn, "reviews", "review_time", "TEXT NOT NULL DEFAULT ''")
    _ensure_column(conn, "reviews", "helpful_votes", "TEXT NOT NULL DEFAULT ''")
    _ensure_column(conn, "reviews", "author", "TEXT NOT NULL DEFAULT ''")
    _ensure_column(conn, "keyword_history", "new_count", "INTEGER DEFAULT 0")
    _ensure_column(conn, "keyword_history", "best_new_count", "INTEGER DEFAULT 0")
    _ensure_column(conn, "keyword_history", "search_count", "INTEGER DEFAULT 1")
    _ensure_column(conn, "keyword_history", "total_at_search", "INTEGER DEFAULT 0")

    conn.commit()
    return conn


def _ensure_column(
    conn: sqlite3.Connection, table: str, column: str, definition: str
) -> None:
    columns = {row[1] for row in conn.execute(f"PRAGMA table_info({table})").fetchall()}
    if column not in columns:
        conn.execute(f"ALTER TABLE {table} ADD COLUMN {column} {definition}")


def get_known_review_ids(conn: sqlite3.Connection, host: str, asin: str) -> set[str]:
    rows = conn.execute(
        "SELECT review_id FROM reviews WHERE host=? AND asin=?", (host, asin)
    ).fetchall()
    return {row[0] for row in rows}


def get_cached_review_count(conn: sqlite3.Connection, host: str, asin: str) -> int:
    row = conn.execute(
        "SELECT COUNT(*) FROM reviews WHERE host=? AND asin=?", (host, asin)
    ).fetchone()
    return row[0] if row else 0


def get_searched_keywords(conn: sqlite3.Connection, host: str, asin: str) -> set[str]:
    """Return set of keywords already searched for this host+asin."""
    rows = conn.execute(
        "SELECT keyword FROM keyword_history WHERE host=? AND asin=?", (host, asin)
    ).fetchall()
    return {row[0] for row in rows}


def get_keyword_history_map(
    conn: sqlite3.Connection, host: str, asin: str
) -> dict[str, dict[str, Any]]:
    rows = conn.execute(
        """SELECT keyword, searched_at, new_count, best_new_count, search_count, total_at_search, job_id
           FROM keyword_history WHERE host=? AND asin=?""",
        (host, asin),
    ).fetchall()
    return {
        row[0]: {
            "keyword": row[0],
            "searched_at": row[1],
            "new_count": row[2],
            "best_new_count": row[3],
            "search_count": row[4],
            "total_at_search": row[5],
            "job_id": row[6],
        }
        for row in rows
    }


def record_keyword_search(
    conn: sqlite3.Connection,
    host: str,
    asin: str,
    keyword: str,
    job_id: str,
    new_count: int,
    total_count: int,
) -> None:
    """Record that a keyword was searched, with results."""
    now = utc_now()
    conn.execute(
        """INSERT INTO keyword_history (
               host, asin, keyword, searched_at, new_count, best_new_count, search_count, total_at_search, job_id
           )
           VALUES (?, ?, ?, ?, ?, ?, 1, ?, ?)
           ON CONFLICT(host, asin, keyword) DO UPDATE SET
               searched_at=excluded.searched_at,
               new_count=excluded.new_count,
               best_new_count=MAX(keyword_history.best_new_count, excluded.best_new_count),
               search_count=keyword_history.search_count + 1,
               total_at_search=excluded.total_at_search,
               job_id=excluded.job_id""",
        (host, asin, keyword, now, new_count, new_count, total_count, job_id),
    )
    conn.commit()


def upsert_reviews(
    conn: sqlite3.Connection,
    host: str,
    asin: str,
    job_id: str,
    rows: list[dict[str, Any]],
) -> int:
    """Insert or update reviews. Returns count of NEW reviews."""
    now = utc_now()
    new_count = 0
    for row in rows:
        review_id = row.get("review_id", "")
        if not review_id:
            continue

        exists = conn.execute(
            "SELECT 1 FROM reviews WHERE host=? AND asin=? AND review_id=?",
            (host, asin, review_id),
        ).fetchone()

        if exists:
            conn.execute(
                """UPDATE reviews SET
                    title=?, body=?, rating_text=?, country_date=?,
                    review_time=?, helpful_votes=?, author=?, review_link=?,
                    source_combo=?, last_seen_at=?, last_job_id=?
                    WHERE host=? AND asin=? AND review_id=?""",
                (
                    row.get("title", ""),
                    row.get("body", ""),
                    row.get("rating_text", ""),
                    row.get("country_date", ""),
                    row.get("review_time", ""),
                    row.get("helpful_votes", ""),
                    row.get("author", ""),
                    row.get("review_link", ""),
                    row.get("source_combo", ""),
                    now,
                    job_id,
                    host,
                    asin,
                    review_id,
                ),
            )
        else:
            conn.execute(
                """INSERT INTO reviews (
                    host, asin, review_id, title, body, rating_text,
                    country_date, review_time, helpful_votes, author,
                    review_link, source_combo, first_seen_at, last_seen_at, last_job_id
                ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
                (
                    host,
                    asin,
                    review_id,
                    row.get("title", ""),
                    row.get("body", ""),
                    row.get("rating_text", ""),
                    row.get("country_date", ""),
                    row.get("review_time", ""),
                    row.get("helpful_votes", ""),
                    row.get("author", ""),
                    row.get("review_link", ""),
                    row.get("source_combo", ""),
                    now,
                    now,
                    job_id,
                ),
            )
            new_count += 1

    conn.commit()
    return new_count


def fetch_cached_reviews(
    conn: sqlite3.Connection, host: str, asin: str
) -> list[dict[str, Any]]:
    """Load all cached reviews for this host+asin."""
    rows = conn.execute(
        """SELECT review_id, title, body, rating_text, country_date,
                  review_time, helpful_votes, author, review_link, source_combo,
                  first_seen_at, last_seen_at
           FROM reviews WHERE host=? AND asin=?
           ORDER BY first_seen_at ASC, review_id ASC""",
        (host, asin),
    ).fetchall()

    results = []
    for row in rows:
        results.append(
            {
                "review_id": row[0],
                "title": row[1],
                "body": row[2],
                "rating_text": row[3],
                "country_date": row[4],
                "review_time": row[5],
                "helpful_votes": row[6],
                "author": row[7],
                "review_link": row[8],
                "source_combo": row[9],
                "first_seen_at": row[10],
                "last_seen_at": row[11],
            }
        )
    return results


def upsert_analysis_records(
    conn: sqlite3.Connection,
    host: str,
    asin: str,
    records: list[dict[str, Any]],
) -> int:
    """Insert or update analysis records. Returns count updated."""
    now = utc_now()
    count = 0
    for rec in records:
        review_id = rec.get("review_id", "")
        if not review_id:
            continue
        conn.execute(
            """INSERT INTO analyses (
                host, asin, review_id, country, rating, review_text,
                translated_text, summary, sentiment, categories, tags,
                focus_marks, updated_at
            ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
            ON CONFLICT(host, asin, review_id) DO UPDATE SET
                country=excluded.country, rating=excluded.rating,
                review_text=excluded.review_text,
                translated_text=excluded.translated_text,
                summary=excluded.summary, sentiment=excluded.sentiment,
                categories=excluded.categories, tags=excluded.tags,
                focus_marks=excluded.focus_marks, updated_at=excluded.updated_at""",
            (
                host,
                asin,
                review_id,
                rec.get("country", ""),
                rec.get("rating", ""),
                rec.get("review_text", ""),
                rec.get("translated_text", ""),
                rec.get("summary", ""),
                rec.get("sentiment", ""),
                rec.get("categories", ""),
                rec.get("tags", ""),
                rec.get("focus_marks", ""),
                now,
            ),
        )
        count += 1
    conn.commit()
    return count


def fetch_cached_analysis_map(
    conn: sqlite3.Connection, host: str, asin: str
) -> dict[str, dict[str, Any]]:
    """Load all analysis records indexed by review_id."""
    rows = conn.execute(
        "SELECT * FROM analyses WHERE host=? AND asin=?", (host, asin)
    ).fetchall()
    col_names = [
        desc[0] for desc in conn.execute("SELECT * FROM analyses LIMIT 1").description
    ]

    result = {}
    for row in rows:
        rec = dict(zip(col_names, row))
        result[rec["review_id"]] = rec
    return result


def create_job(
    conn: sqlite3.Connection,
    job_id: str,
    host: str,
    asin: str,
    source_url: str,
) -> None:
    conn.execute(
        """INSERT INTO jobs (job_id, host, asin, source_url, started_at, status)
           VALUES (?, ?, ?, ?, ?, 'running')""",
        (job_id, host, asin, source_url, utc_now()),
    )
    conn.commit()


def finish_job(
    conn: sqlite3.Connection,
    job_id: str,
    status: str,
    *,
    current_seen_count: int = 0,
    new_review_count: int = 0,
    cached_total: int = 0,
    stats: list[dict[str, Any]] | None = None,
) -> None:
    try:
        conn.execute(
            """UPDATE jobs SET
                finished_at=?, status=?, current_seen_count=?,
                new_review_count=?, cached_total=?, stats_json=?
                WHERE job_id=?""",
            (
                utc_now(),
                status,
                current_seen_count,
                new_review_count,
                cached_total,
                json.dumps(stats or [], ensure_ascii=False),
                job_id,
            ),
        )
        conn.commit()
    except Exception:
        pass  # best-effort, don't mask original error


def cleanup_stale_jobs(conn: sqlite3.Connection, *, max_age_minutes: int = 180) -> int:
    """Mark jobs running longer than max_age_minutes as 'interrupted'."""
    from datetime import timedelta

    cutoff = datetime.now(timezone.utc) - timedelta(minutes=max_age_minutes)
    cutoff_str = cutoff.strftime("%Y-%m-%dT%H:%M:%S.%fZ")
    rows = conn.execute(
        "SELECT job_id FROM jobs WHERE status='running' AND started_at < ?",
        (cutoff_str,),
    ).fetchall()
    stale_ids = [row[0] for row in rows]
    if stale_ids:
        conn.executemany(
            "UPDATE jobs SET finished_at=?, status='interrupted' WHERE job_id=?",
            [(utc_now(), jid) for jid in stale_ids],
        )
        conn.commit()
    return len(stale_ids)


def export_analysis_to_records(
    conn: sqlite3.Connection, host: str, asin: str
) -> list[dict[str, Any]]:
    """Export full analysis records (reviews JOIN analyses) for delivery."""
    rows = conn.execute(
        """SELECT r.review_id, r.review_link, r.review_time, r.helpful_votes,
                  r.author, r.country_date, r.rating_text, r.body,
                  a.country, a.rating, a.review_text, a.translated_text,
                  a.summary, a.sentiment, a.categories, a.tags, a.focus_marks
           FROM reviews r
           LEFT JOIN analyses a ON a.host=r.host AND a.asin=r.asin AND a.review_id=r.review_id
           WHERE r.host=? AND r.asin=?
           ORDER BY r.first_seen_at ASC, r.review_id ASC""",
        (host, asin),
    ).fetchall()

    records = []
    for i, row in enumerate(rows, start=1):
        records.append(
            {
                "序号": str(i),
                "评论用户名": row[4] or "",
                "国家": row[8] or "",
                "星级评分": row[9] or (row[6] or ""),
                "评论原文": row[7] or "",
                "评论中文版": row[11] or "",
                "评论概括": row[12] or "",
                "情感倾向": row[13] or "",
                "类别分类": row[14] or "",
                "标签": row[15] or "",
                "重点标记": row[16] or "",
                "评论链接网址": row[1] or "",
                "评论时间": row[2] or "",
                "评论点赞数": row[3] or "",
                "review_id": row[0],
            }
        )
    return records


path: scripts/review_delivery_schema.py
content: from __future__ import annotations

import json
import math
import re
from pathlib import Path
from typing import Any

import pandas as pd
from openpyxl import load_workbook
from openpyxl.styles import Alignment, Font

DELIVERY_COLUMNS = [
    "序号",
    "评论用户名",
    "国家",
    "星级评分",
    "评论原文",
    "评论中文版",
    "评论概括",
    "情感倾向",
    "类别分类",
    "标签",
    "重点标记",
    "评论链接网址",
    "评论时间",
    "评论点赞数",
]

REQUIRED_ANALYSIS_COLUMNS = [
    "评论原文",
    "评论中文版",
    "评论概括",
    "情感倾向",
    "类别分类",
    "标签",
    "重点标记",
]

GUIDE_ROWS = [
    ("序号", "从 1 开始连续编号。"),
    ("评论用户名", "评论者的 Amazon 用户名；没有来源就留空。"),
    ("国家", "页面明确给出的国家/地区；没有来源就留空。"),
    ("星级评分", "仅在页面有显式星级时填写。"),
    ("评论原文", "保留原语言原文；若正文为空，可回退到标题。"),
    ("评论中文版", "忠实翻译，不额外润色。"),
    ("评论概括", "一句话概括评论核心观点。"),
    ("情感倾向", "只允许 Positive / Negative / Neutral。"),
    ("类别分类", "通用运营分类口径，默认 1-2 个高信号类别。"),
    ("标签", "短标题型标签，单个标签不超过 10 个字。"),
    ("重点标记", "优先标出性价比、质量、功能改进建议、竞品/型号等。"),
    ("评论链接网址", "可追溯到具体评论的原始链接。"),
    ("评论时间", "评论页面明确给出的时间；没有来源就留空。"),
    ("评论点赞数", "评论页面明确给出的 helpful/useful 数；没有来源就留空。"),
]

QUALITY_ROWS = [
    ("标签长度", "标签列中的单个标签不超过 10 个字。"),
    ("情感值", "必须是 Positive / Negative / Neutral。"),
    ("来源完整性", "国家、评分、时间、点赞数只在页面明确提供时填写。"),
    ("同类合并", "尽量把同义表达收敛为少量稳定标签。"),
    ("重点洞察", "优先标出性价比、质量、功能改进建议、竞品/型号。"),
]

FIELD_ALIASES = {
    "序号": "序号",
    "seq": "序号",
    "index": "序号",
    "id": "序号",
    "评论用户名": "评论用户名",
    "author": "评论用户名",
    "reviewer": "评论用户名",
    "reviewer_name": "评论用户名",
    "user_name": "评论用户名",
    "username": "评论用户名",
    "国家": "国家",
    "country": "国家",
    "country_cn": "国家",
    "region": "国家",
    "market": "国家",
    "星级评分": "星级评分",
    "rating": "星级评分",
    "star_rating": "星级评分",
    "rating_text": "星级评分",
    "评论原文": "评论原文",
    "review_text": "评论原文",
    "body": "评论原文",
    "title": "评论原文",
    "评论中文版": "评论中文版",
    "translated_text": "评论中文版",
    "translated_cn": "评论中文版",
    "评论概括": "评论概括",
    "summary": "评论概括",
    "summary_cn": "评论概括",
    "情感倾向": "情感倾向",
    "sentiment": "情感倾向",
    "类别分类": "类别分类",
    "category": "类别分类",
    "categories": "类别分类",
    "标签": "标签",
    "tag": "标签",
    "tags": "标签",
    "tags_cn": "标签",
    "重点标记": "重点标记",
    "focus_mark": "重点标记",
    "focus_marks": "重点标记",
    "priority_mark": "重点标记",
    "评论链接网址": "评论链接网址",
    "review_link": "评论链接网址",
    "评论时间": "评论时间",
    "review_time": "评论时间",
    "country_date": "评论时间",
    "评论点赞数": "评论点赞数",
    "helpful_votes": "评论点赞数",
    "helpful_vote": "评论点赞数",
    "helpful_vote_statement": "评论点赞数",
}

SENTIMENT_MAP = {
    "positive": "Positive",
    "pos": "Positive",
    "正向": "Positive",
    "积极": "Positive",
    "negative": "Negative",
    "neg": "Negative",
    "负向": "Negative",
    "消极": "Negative",
    "neutral": "Neutral",
    "neu": "Neutral",
    "中性": "Neutral",
    "一般": "Neutral",
}

ALLOWED_CATEGORIES = [
    "Praise on product",
    "Questions/Worrying",
    "Negative emotions",
    "Suggestion",
    "Nothing particular",
    "Being supportive to the brand",
    "Expecting",
    "Being Sarcastic",
    "Competitor comparison",
    "Status update",
]

CATEGORY_ALIASES = {
    "Priase on product": "Praise on product",
    "Product praise": "Praise on product",
    "Questions / Worrying": "Questions/Worrying",
    "Question": "Questions/Worrying",
    "Concern": "Questions/Worrying",
}

FOCUS_MARK_EXPANSIONS = {
    "品牌/售后": ["品牌", "售后"],
    "物流/包装": ["物流", "包装"],
    "竞品/型号": ["竞品", "型号"],
    "安装/适配": ["安装", "适配"],
    "软件/设置": ["软件", "设置"],
    "配件/兼容性": ["配件", "兼容性"],
}

FOCUS_MARK_MAP = {
    "value_for_money": "性价比",
    "quality": "质量",
    "feature_suggestion": "功能改进建议",
    "competitor_model": "竞品/型号",
    "software_setup": "软件/设置",
    "installation_fit": "安装/适配",
    "brand_after_sales": "品牌/售后",
    "logistics_packaging": "物流/包装",
    "accessory_compatibility": "配件/兼容性",
}

TAG_SHORTCUTS = {
    "用户整体满意": "整体满意",
    "对产品整体满意": "整体满意",
    "对品牌整体信任": "品牌信任",
    "产品质量不错": "质量不错",
    "产品质量一般": "质量一般",
    "产品质量较差": "质量较差",
    "建议改进功能": "功能建议",
    "软件设置复杂": "设置复杂",
    "包装到货完好": "包装完好",
}

TAG_SEPARATORS = ("；", ";", "、", "|", "，", ",")


def normalize_space(value: Any) -> str:
    return re.sub(r"\s+", " ", str(value or "")).strip()


NESTED_RECORD_KEYS = (
    "machine",
    "delivery",
    "fields",
    "analysis",
    "semantic",
    "payload",
    "data",
    "row",
    "record",
)


def flatten_record(raw: Any) -> dict[str, Any]:
    if not isinstance(raw, dict):
        return {}

    flattened: dict[str, Any] = {}
    for nested_key in NESTED_RECORD_KEYS:
        nested = raw.get(nested_key)
        if isinstance(nested, dict):
            flattened.update(flatten_record(nested))

    for key, value in raw.items():
        if key in NESTED_RECORD_KEYS and isinstance(value, dict):
            continue
        flattened[key] = value

    return flattened


def normalize_key(key: Any) -> str:
    text = normalize_space(key)
    lowered = text.lower()
    return FIELD_ALIASES.get(text, FIELD_ALIASES.get(lowered, text))


def split_values(raw: Any) -> list[str]:
    value = normalize_space(raw)
    if not value:
        return []
    for separator in TAG_SEPARATORS[1:]:
        value = value.replace(separator, TAG_SEPARATORS[0])
    return [item.strip() for item in value.split(TAG_SEPARATORS[0]) if item.strip()]


def dedupe_preserve_order(values: list[str]) -> list[str]:
    seen: set[str] = set()
    result: list[str] = []
    for value in values:
        if value not in seen:
            seen.add(value)
            result.append(value)
    return result


def shorten_tag(tag: str) -> str:
    candidate = normalize_space(tag).replace("“", "").replace("”", "")
    candidate = TAG_SHORTCUTS.get(candidate, candidate)
    if len(candidate) <= 10:
        return candidate
    for old, new in [
        ("用户", ""),
        ("产品", ""),
        ("设备", ""),
        ("整体", ""),
        ("非常", ""),
        ("明显", ""),
        ("需要", ""),
    ]:
        candidate = candidate.replace(old, new)
        if len(candidate) <= 10:
            return candidate
    return candidate[:10]


def normalize_sentiment(raw: Any) -> str:
    value = normalize_space(raw)
    if not value:
        return ""
    return SENTIMENT_MAP.get(
        value.lower(),
        value if value in {"Positive", "Negative", "Neutral"} else "Neutral",
    )


def normalize_categories(raw: Any) -> str:
    normalized: list[str] = []
    for item in split_values(str(raw).replace(" / ", "；").replace("/", "；")):
        candidate = CATEGORY_ALIASES.get(item, item)
        if candidate in ALLOWED_CATEGORIES and candidate not in normalized:
            normalized.append(candidate)
    if not normalized:
        return "Nothing particular"
    return " / ".join(normalized[:2])


def normalize_tags(raw: Any) -> str:
    cleaned: list[str] = []
    value = str(raw or "").replace("/", "；")
    for tag in split_values(value):
        lowered = tag.lower()
        if lowered in {"none", "null", "n/a", "na"} or tag in {"", "???", "无"}:
            continue
        cleaned.append(shorten_tag(tag))
    return "；".join(dedupe_preserve_order(cleaned))


def normalize_focus_marks(raw: Any) -> str:
    if not raw:
        return ""
    values = split_values(raw)
    expanded: list[str] = []
    for value in values:
        expanded.extend(FOCUS_MARK_EXPANSIONS.get(value, [value]))
    normalized: list[str] = []
    for value in expanded:
        lowered = value.lower()
        if lowered in {"none", "null", "n/a", "na"} or value in {"", "???", "无"}:
            continue
        mapped = FOCUS_MARK_MAP.get(lowered, value)
        normalized.append(mapped)
    return "；".join(dedupe_preserve_order(normalized))


MONTH_MAP = {
    "january": 1,
    "february": 2,
    "march": 3,
    "april": 4,
    "may": 5,
    "june": 6,
    "july": 7,
    "august": 8,
    "september": 9,
    "october": 10,
    "november": 11,
    "december": 12,
    "jan": 1,
    "feb": 2,
    "mar": 3,
    "apr": 4,
    "jun": 6,
    "jul": 7,
    "aug": 8,
    "sep": 9,
    "oct": 10,
    "nov": 11,
    "dec": 12,
}


def normalize_review_time(raw: Any) -> str:
    value = normalize_space(raw)
    if not value:
        return ""
    for separator in (" on ", " il ", " am ", " le ", " el ", " en ", " em "):
        if separator in value:
            value = value.rsplit(separator, 1)[-1].strip()

    # YYYY年M月D日
    match = re.search(r"(\d{4})年(\d{1,2})月(\d{1,2})日", value)
    if match:
        y, m, d = match.group(1), match.group(2).zfill(2), match.group(3).zfill(2)
        return f"{y}-{m}-{d}"

    # DD Month YYYY or D Month YY
    match = re.search(r"(\d{1,2})\s+([A-Za-zÀ-ÿ]+)\s+(\d{2,4})", value)
    if match:
        day = int(match.group(1))
        month_str = match.group(2).lower()
        year_str = match.group(3)
        year = int(year_str) if len(year_str) == 4 else (2000 + int(year_str))
        month = MONTH_MAP.get(month_str)
        if month:
            return f"{year}-{month:02d}-{day:02d}"

    # YYYY-MM-DD
    match = re.search(r"(\d{4})-(\d{2})-(\d{2})", value)
    if match:
        return f"{match.group(1)}-{match.group(2)}-{match.group(3)}"

    # DD/MM/YYYY or DD.MM.YYYY
    match = re.search(r"(\d{1,2})[/.](\d{1,2})[/.](\d{4})", value)
    if match:
        d, m, y = match.group(1).zfill(2), match.group(2).zfill(2), match.group(3)
        return f"{y}-{m}-{d}"

    # Relative time: "X days/weeks/months ago"
    match = re.search(r"(\d+)\s+(day|week|month|year)s?\s+ago", value, re.I)
    if match:
        from datetime import datetime, timedelta

        amount = int(match.group(1))
        unit = match.group(2).lower()
        now = datetime.now()
        if unit == "day":
            delta = timedelta(days=amount)
        elif unit == "week":
            delta = timedelta(weeks=amount)
        elif unit == "month":
            delta = timedelta(days=amount * 30)
        else:
            delta = timedelta(days=amount * 365)
        target = now - delta
        return target.strftime("%Y-%m-%d")

    return ""


def normalize_helpful_votes(raw: Any) -> str:
    value = normalize_space(raw)
    if not value:
        return ""
    lowered = value.lower()
    if "one person" in lowered or "one customer" in lowered:
        return "1"
    match = re.search(r"(\d[\d,]*)", lowered)
    if match:
        return match.group(1).replace(",", "")
    return ""


def build_review_link(review_id: str, host: str = "", raw_link: Any = "") -> str:
    link = normalize_space(raw_link)
    if link:
        return link
    review_id = normalize_space(review_id)
    host = normalize_space(host)
    if not review_id or not host:
        return ""
    base = host if host.startswith("http") else f"https://{host}"
    return f"{base.rstrip('/')}/gp/customer-reviews/{review_id}/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8"


def parse_number_value(value: Any) -> int | float | None:
    text = normalize_space(value).replace(",", "")
    if not text:
        return None
    try:
        numeric = float(text)
    except ValueError:
        return None
    if numeric.is_integer():
        return int(numeric)
    return numeric


def normalize_delivery_record(raw: dict[str, Any], index: int) -> dict[str, str]:
    raw = flatten_record(raw)
    record = {column: "" for column in DELIVERY_COLUMNS}
    host = normalize_space(raw.get("host") or raw.get("site") or raw.get("source_host"))
    primary_review_text = normalize_space(
        raw.get("评论原文")
        or raw.get("review_text")
        or raw.get("body")
        or raw.get("content")
        or raw.get("review_body")
        or raw.get("title")
        or raw.get("review_title")
    )

    for key, value in raw.items():
        mapped_key = normalize_key(key)
        if mapped_key not in record:
            continue
        if mapped_key == "序号":
            record[mapped_key] = normalize_space(value)
        elif mapped_key == "评论链接网址":
            record[mapped_key] = build_review_link(
                str(raw.get("review_id") or raw.get("reviewId") or ""),
                host,
                value,
            )
        elif mapped_key == "评论时间":
            record[mapped_key] = normalize_review_time(value)
        elif mapped_key == "评论点赞数":
            record[mapped_key] = normalize_helpful_votes(value)
        else:
            record[mapped_key] = normalize_space(value)

    if primary_review_text:
        record["评论原文"] = primary_review_text
    record["序号"] = record["序号"] or str(index)
    record["情感倾向"] = normalize_sentiment(record["情感倾向"])
    record["类别分类"] = normalize_categories(record["类别分类"])
    record["标签"] = normalize_tags(record["标签"])
    record["重点标记"] = normalize_focus_marks(record["重点标记"])
    record["评论时间"] = normalize_review_time(record["评论时间"])
    record["评论点赞数"] = normalize_helpful_votes(record["评论点赞数"])
    if not record["评论链接网址"]:
        record["评论链接网址"] = build_review_link(
            str(raw.get("review_id") or raw.get("reviewId") or ""),
            host,
            raw.get("review_link") or raw.get("评论链接网址") or "",
        )
    return record


def build_delivery_records(records: list[dict[str, Any]]) -> list[dict[str, str]]:
    return [
        normalize_delivery_record(record, index)
        for index, record in enumerate(records, start=1)
    ]


def validate_delivery_records(
    records: list[dict[str, str]], *, strict: bool = False
) -> list[str]:
    warnings: list[str] = []
    for row_index, record in enumerate(records, start=1):
        for column in REQUIRED_ANALYSIS_COLUMNS:
            if strict and not record.get(column):
                warnings.append(f"第 {row_index} 行缺少必填字段：{column}")
        sentiment = record.get("情感倾向", "")
        if sentiment and sentiment not in {"Positive", "Negative", "Neutral"}:
            warnings.append(f"第 {row_index} 行情感倾向非法：{sentiment}")
        for field in ("标签",):
            for tag in split_values(record.get(field, "")):
                if len(tag) > 10:
                    warnings.append(f"第 {row_index} 行 {field} 超过 10 个字：{tag}")

    if strict and warnings:
        raise ValueError("\n".join(warnings))
    return warnings


def build_delivery_dataframe(records: list[dict[str, str]]) -> pd.DataFrame:
    return pd.DataFrame(records, columns=DELIVERY_COLUMNS)


def write_delivery_workbook(df: pd.DataFrame, output_xlsx: Path) -> None:
    guide_df = pd.DataFrame(GUIDE_ROWS, columns=["字段", "说明"])
    quality_df = pd.DataFrame(QUALITY_ROWS, columns=["检查项", "要求"])

    with pd.ExcelWriter(output_xlsx, engine="openpyxl") as writer:
        df.to_excel(writer, index=False, sheet_name="comments")
        guide_df.to_excel(writer, index=False, sheet_name="guide")
        quality_df.to_excel(writer, index=False, sheet_name="quality")

    workbook = load_workbook(output_xlsx)
    for sheet_name in ("comments", "guide", "quality"):
        worksheet = workbook[sheet_name]
        worksheet.freeze_panes = "A2"
        for row in worksheet.iter_rows():
            for cell in row:
                cell.alignment = Alignment(wrap_text=True, vertical="top")
        for cell in worksheet[1]:
            cell.font = Font(bold=True)

    widths = {
        "A": 8,  # 序号
        "B": 18,  # 评论用户名
        "C": 14,  # 国家
        "D": 10,  # 星级评分
        "E": 48,  # 评论原文
        "F": 48,  # 评论中文版
        "G": 24,  # 评论概括
        "H": 12,  # 情感倾向
        "I": 24,  # 类别分类
        "J": 18,  # 标签
        "K": 24,  # 重点标记
        "L": 40,  # 评论链接网址
        "M": 20,  # 评论时间
        "N": 16,  # 评论点赞数
    }
    comments_sheet = workbook["comments"]
    for column, width in widths.items():
        comments_sheet.column_dimensions[column].width = width

    workbook["guide"].column_dimensions["A"].width = 18
    workbook["guide"].column_dimensions["B"].width = 80
    workbook["quality"].column_dimensions["A"].width = 18
    workbook["quality"].column_dimensions["B"].width = 80
    workbook.save(output_xlsx)


def write_delivery_artifacts(
    records: list[dict[str, Any]],
    output_json: Path,
    output_xlsx: Path,
    output_csv: Path,
    *,
    strict: bool = False,
) -> dict[str, Any]:
    normalized = build_delivery_records(records)
    warnings = validate_delivery_records(normalized, strict=strict)

    output_json.parent.mkdir(parents=True, exist_ok=True)
    output_xlsx.parent.mkdir(parents=True, exist_ok=True)
    output_csv.parent.mkdir(parents=True, exist_ok=True)

    output_json.write_text(
        json.dumps(normalized, ensure_ascii=False, indent=2), encoding="utf-8"
    )
    dataframe = build_delivery_dataframe(normalized)
    write_delivery_workbook(dataframe, output_xlsx)
    dataframe.to_csv(output_csv, index=False, encoding="utf-8-sig")

    return {
        "rows": len(normalized),
        "warnings": warnings,
        "records": normalized,
    }


Security scan verdicts — ClawScan: suspicious | static analysis: suspicious | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)