Skill: skernelx/mysearch (version 0.1.11)

--- name: mysearch version: "0.1.11" description: >- DEFAULT search skill for OpenClaw. Aggregates Tavily, Firecrawl, Exa, and optional X/social search behind one search path. Use for ANY external lookup that needs current web results, docs, GitHub, changelog, pricing, URL extraction, or X discussion. Prefer this over legacy Tavily-only skill or raw web_search when MySearch is healthy. author: skernelx license: MIT repository: https://github.com/skernelx/MySearch-Proxy homepage: https://github.com/skernelx/MySearch-Proxy/tree/main/openclaw security_disclosure: | MySearch sends queries and whichever provider or proxy credentials you configure to the configured endpoints for Tavily, Firecrawl, Exa, and optional xAI/social search. Prefer configuring env vars through OpenClaw skill config instead of copying a .env file into an installed skill folder. Only point MYSEARCH_PROXY_BASE_URL at a host you trust. The bundled installer only copies local files from this bundle and does not download remote code or modify other skills. metadata: clawdbot: emoji: "🔎" requires: bins: - bash - python3 openclaw: emoji: "🔎" requires: bins: - bash - python3 env: - MYSEARCH_PROXY_BASE_URL - MYSEARCH_PROXY_API_KEY - MYSEARCH_TAVILY_API_KEY - MYSEARCH_FIRECRAWL_API_KEY primaryEnv: MYSEARCH_PROXY_API_KEY tags: - search - web - docs - tavily - firecrawl - exa - social --- # MySearch For OpenClaw MySearch 是给 OpenClaw 用的默认搜索 skill。 如果你是 AI 助手，并且用户只是给了仓库地址或 `openclaw/` 目录： - 先打开 `openclaw/README.md` - 先按 `README` 完成安装与验收 - 再回到这个 `SKILL.md` 执行搜索规则和调用策略 它把 Tavily、Firecrawl、Exa、X / Social 聚合成同一个入口，并且按任务类型自动路由： - 最新网页、普通实时搜索：优先 Tavily - Tavily 结果不够时，网页发现可补 Exa - 文档、GitHub、pricing、changelog、PDF：优先 Firecrawl - X / Twitter / 社交舆情：优先 xAI 或 compatible `/social/search` - 单页正文抓取：优先 Firecrawl，失败或空正文时回退 Tavily extract ## 最小配置 Hub 版 skill 已经自带 runtime，不需要在安装时再下载远程代码。 推荐最小配置： - `MYSEARCH_PROXY_BASE_URL` - `MYSEARCH_PROXY_API_KEY` 这两项配好后： - `Tavily / Firecrawl / Exa` 会默认都走统一 proxy - 如果 proxy 同时接通了 `Social / X`，这一套 token 也会继续复用 - OpenClaw 侧不需要再分别维护三套 provider token 兼容旧接法时，仍可直接填： - `MYSEARCH_TAVILY_API_KEY` - `MYSEARCH_FIRECRAWL_API_KEY` 可选增强： - `MYSEARCH_XAI_API_KEY` - `MYSEARCH_XAI_BASE_URL` - `MYSEARCH_XAI_SOCIAL_BASE_URL` - `MYSEARCH_XAI_SEARCH_MODE=official|compatible` 如果没有 X / Social 配置，MySearch 仍然可以正常完成： - `web` - `news` - `docs` - `github` - `pdf` - `extract` - `research` 只有 `mode="social"` 或 `--include-social` 才会要求 X / Social。 ## OpenClaw 配置建议 优先把统一 proxy 配进 OpenClaw skill env，而不是到处复制 provider key 或 shell 环境。 `MYSEARCH_PROXY_BASE_URL` 只应该指向你自己部署或明确可信的 proxy。 `mysearch_openclaw.py` 会优先读取 `openclaw.json` 里的 `skills.entries.mysearch.env`，正式部署不需要依赖 `.env`。 ```json { "skills": { "entries": { "mysearch": { "enabled": true, "env": { "MYSEARCH_PROXY_BASE_URL": "https://search.hunters.works", "MYSEARCH_PROXY_API_KEY": "mysp-..." } } } } } ``` 如果你暂时没有统一 proxy，再退回旧接法： ```json { "skills": { "entries": { "mysearch": { "enabled": true, "env": { "MYSEARCH_TAVILY_API_KEY": "tvly-...", "MYSEARCH_FIRECRAWL_API_KEY": "fc-..." } } } } } ``` 只有在你直接调试这个仓库工作树时，才建议复制 `.env.example` 到本地 `.env`。 Hub 安装或正式 OpenClaw 部署优先用上面的 skill env 注入，不要默认把 secrets 复制进已安装的 skill 目录。 本地调试示例： ```bash cp {baseDir}/.env.example {baseDir}/.env python3 {baseDir}/scripts/mysearch_openclaw.py health ``` 如果要把 skill 复制到别的 OpenClaw skills 目录，再执行： ```bash bash {baseDir}/scripts/install_openclaw_skill.sh --install-to ~/.openclaw/skills/mysearch ``` ## MySearch-First 规则 只要 `health` 显示至少有可用搜索 provider： - 外部搜索任务优先走 MySearch - 不要把 raw `web_search` 当主流程 - 不要优先走旧的 Tavily-only skill 只有这些情况才回退： - MySearch 还没配置最小 key - 用户明确要求改用别的搜索方式 - MySearch 返回冲突结果，需要额外复核 ## 严格参数规则 `search` / `research` 的 `mode` 只允许： - `auto` - `web` - `news` - `social` - `docs` - `research` - `github` - `pdf` 禁止事项： - 不要发明 `mode="hybrid"` - `hybrid` 只是某些返回结果形态，不是可传参数 - 同时要网页和 X 时，优先： - `--sources web,x` - 或拆成 `social + news` ## 常用命令 ### 健康检查 ```bash python3 {baseDir}/scripts/mysearch_openclaw.py health ``` ### 普通网页搜索 ```bash python3 {baseDir}/scripts/mysearch_openclaw.py search \ --query "best search MCP server" \ --mode web ``` ### 今天 X 上在热议什么 ```bash python3 {baseDir}/scripts/mysearch_openclaw.py search \ --query "today's biggest stories on X" \ --mode social \ --intent status ``` 规则： - 先 `social` - 不要先跑 `news` - 不要先混用 raw `web_search` ### 今天 X 热议 + 网页新闻一起对照 单次： ```bash python3 {baseDir}/scripts/mysearch_openclaw.py search \ --query "today's biggest stories on X" \ --sources web,x \ --intent status \ --strategy verify ``` 或者双次： ```bash python3 {baseDir}/scripts/mysearch_openclaw.py search --query "..." --mode social --intent status python3 {baseDir}/scripts/mysearch_openclaw.py search --query "..." --mode news --intent status ``` 输出时必须区分： - X 上在热议什么 - 网页新闻在报道什么 ### 文档 / GitHub / pricing / changelog ```bash python3 {baseDir}/scripts/mysearch_openclaw.py search \ --query "OpenAI responses API pricing" \ --mode docs \ --intent resource ``` ### 抓正文 ```bash python3 {baseDir}/scripts/mysearch_openclaw.py extract \ --url "https://example.com/post" ``` ### 小型研究包 ```bash python3 {baseDir}/scripts/mysearch_openclaw.py research \ --query "best search MCP server 2026" \ --intent exploratory \ --include-social ``` ## 输出要求 - 优先给结论，再给来源 - 保留 URL - 区分事实、引文和推断 - 同时包含网页和 X 时，明确分区，不要混成一句模糊总结 - `max_results` 默认保持小一些，先拿 3 到 5 条

path: README.md
content: # MySearch OpenClaw Skill

[English Guide](./README_EN.md) · [返回仓库](../README.md)

`openclaw/` 是给 OpenClaw 和 ClawHub 准备的独立 skill bundle。

这一层和 `skill/` 的区别很明确：

- `skill/`
  - 主要负责让 Codex / Claude Code 知道怎么安装与使用 MySearch
- `openclaw/`
  - 负责给 OpenClaw 提供真正可安装的 skill bundle

这个 bundle 自带 runtime，不需要再额外下载远端代码。

## 适合什么时候用

- 你想把 MySearch 装进 OpenClaw
- 你想从 ClawHub 安装或更新
- 你想在本地仓库里直接替换 skill
- 你希望 OpenClaw 也走统一的 `proxy-first` 搜索链

公开页面：

- ClawHub：
  [clawhub.ai/skernelx/mysearch](https://clawhub.ai/skernelx/mysearch)

## 当前推荐配置

最推荐的是只给 skill 注入两项环境变量：

```env
MYSEARCH_PROXY_BASE_URL=https://your-mysearch-proxy.example.com
MYSEARCH_PROXY_API_KEY=mysp-...
```

这样做的好处：

- Tavily / Firecrawl / Exa 默认共用同一个 token
- 如果 Proxy 已接通 Social / X，也可以继续走同一套 token
- OpenClaw 侧配置最少
- 不需要把一堆 provider key 再散到 skill 里

如果你暂时没有 Proxy，再回退到直连 provider：

```env
MYSEARCH_TAVILY_API_KEY=tvly-...
MYSEARCH_FIRECRAWL_API_KEY=fc-...
MYSEARCH_EXA_API_KEY=exa-...
MYSEARCH_XAI_API_KEY=xai-...
```

## 安装方式

### 方式 A：从 ClawHub 安装

如果你的环境已经接好了 ClawHub，这是最省心的方式。

技能页面：

- [clawhub.ai/skernelx/mysearch](https://clawhub.ai/skernelx/mysearch)

安装后重点不是改 bundle 文件，而是给 OpenClaw skill config 注入 env。

`mysearch_openclaw.py` 现在会优先读取 OpenClaw 的 `openclaw.json` 里
`skills.entries.mysearch.env`，所以正式部署不需要把 key 再复制进 skill 目录。

推荐做法：

- 在 OpenClaw 的 skill 配置里填 `MYSEARCH_PROXY_BASE_URL`
- 再填 `MYSEARCH_PROXY_API_KEY`
- 不要把 secret 明文写进仓库文件

### 方式 B：从本地仓库安装 bundle

如果你正在本地调试，或者就是想直接覆盖掉现有 skill，执行：

```bash
bash openclaw/scripts/install_openclaw_skill.sh \
  --install-to ~/.openclaw/skills/mysearch
```

这个脚本会：

- 复制 `openclaw/` skill bundle
- 保留 bundled runtime
- 带上 `.env.example`
- 不下载远端代码
- 不修改其他 skill

## 推荐的 OpenClaw skill env 注入方式

优先通过 OpenClaw 的 skill env 配置注入，不要把 secret 放进安装目录。

推荐最小配置：

```json
{
  "skills": {
    "entries": {
      "mysearch": {
        "enabled": true,
        "env": {
          "MYSEARCH_PROXY_BASE_URL": "https://your-mysearch-proxy.example.com",
          "MYSEARCH_PROXY_API_KEY": "mysp-..."
        }
      }
    }
  }
}
```

如果你确实在本地调试 bundle，也可以：

```bash
cp openclaw/.env.example openclaw/.env
```

但这只建议用于本地调试，不建议作为正式部署方式。

## 安装后怎么验收

优先跑健康检查：

```bash
python3 ~/.openclaw/skills/mysearch/scripts/mysearch_openclaw.py health
```

再跑一轮搜索 smoke test：

```bash
python3 ~/.openclaw/skills/mysearch/scripts/mysearch_openclaw.py search \
  --query "OpenAI latest announcements" \
  --mode web
```

如果你要测文档搜索：

```bash
python3 ~/.openclaw/skills/mysearch/scripts/mysearch_openclaw.py search \
  --query "OpenAI Responses API docs" \
  --mode docs \
  --intent resource
```

如果你配置了 Social / X，再补：

```bash
python3 ~/.openclaw/skills/mysearch/scripts/mysearch_openclaw.py search \
  --query "Model Context Protocol" \
  --mode social \
  --intent status
```

## 这个 skill 到底提供了什么

OpenClaw 版 MySearch 本质上还是同一套能力，只是打成了 skill bundle：

- `search`
- `extract`
- `research`
- `health`

默认理解方式：

- `web / news`
  - 优先 Tavily
- `docs / github / pdf`
  - 优先 Firecrawl
- 普通网页补充发现
  - 可回退 Exa
- `social`
  - 走 xAI 或兼容 `/social/search`

## 版本与优化说明（v0.1.11）

`mysearch@0.1.11` 已同步以下运行时优化：

- 配置入口收口：
  - runtime 现在会优先读取宿主配置里的 skill env，不再默认把 `.env` 当主入口。
  - `mysearch_openclaw.py` 会优先读取 `openclaw.json` 中 `skills.entries.mysearch.env`。
  - `.env` 仍可用，但只建议本地 bundle 调试时使用。
  - 读取宿主配置时不再强依赖 Python 3.11 的 `tomllib`，Python 3.10 也能直接跑。

- 文档结果质量：
  - `docs / github / pdf / resource / tutorial` 的混合结果现在会优先官方文档域名与文档路径。
  - 显式传 `include_domains` 时，命中域名的结果会更稳定地排在前面。
  - `reddit / arxiv / researchgate / medium / youtube` 这类第三方页面会被后置，不再轻易抢前排。
  - `citations` 会跟随重排后的结果顺序，避免正文结果和引用顺序打架。
  - Firecrawl 若在 `site:` 域名约束查询下回空，会自动补一轮无 `site:` 的 Firecrawl 检索并在本地按域名过滤，减少 docs 查询被迫切到 Tavily 的概率。

- Provider 健康与容错：
  - `health` 现在会直接显示 provider 的 `live_status` 与 `live_error`，不再只报 `available_keys`。
  - 若某个 provider 已配置但 key 实际失效，例如 `Tavily 401 deactivated`，wrapper 会明确显示 `auth_error`。
  - docs / resource 路由和 Firecrawl 域名 fallback 会自动跳过处于 `auth_error` 的 Tavily，避免把“key 已失效”误判成“没配 provider”。

- 路由稳定性：
  - 显式指定 provider 时，不再被 balanced 策略自动混成 hybrid。
- 抓取质量：
  - `extract` 新增更多假正文识别。
  - `auto` 模式可识别验证码页、GitHub blob 页面壳，并针对公开 GitHub blob 优先抓 raw 正文。
- 社交结果一致性：
  - 日期过滤后若没有命中，`results / citations / answer` 会保持一致。
- Social / X fallback：
  - `social/search` 支持主模型结果太少或上游失败时自动 fallback。
  - 返回新增 `route` 字段，可直接查看主模型、fallback 模型和每轮尝试结果。

- 并行执行：
  - 混合检索与 `research` 分支支持并行调用，减少总耗时。
- 缓存能力：
  - `search` 与 `extract` 增加 TTL 缓存。
- 可观测性：
  - 返回中新增 `route_debug` 与 `cache` 字段，便于定位路由和命中行为。
- 健康信息增强：
  - `health` 返回新增 `runtime`、`routing_defaults`、`cache`。

配套环境变量：

```env
MYSEARCH_MAX_PARALLEL_WORKERS=4
MYSEARCH_SEARCH_CACHE_TTL_SECONDS=30
MYSEARCH_EXTRACT_CACHE_TTL_SECONDS=300
```

注意：

- 若通过 `mysearch_openclaw.py` 每次单独执行命令，属于一次一进程，缓存不会跨命令持久。
- 在 OpenClaw 常驻会话中，缓存才会体现连续命中。

## 安全建议

这份 skill 的推荐安全边界很明确：

- 优先通过 OpenClaw skill env 注入 secret
- 不要把正式 token 复制进 skill 目录
- 不要把 `MYSEARCH_PROXY_BASE_URL` 指向不可信主机
- 如果你直连 provider，也尽量把 key 放到 OpenClaw 配置层，而不是 bundle 文件里

`public.json` 里也保留了对应的公开安全说明，方便 ClawHub 校验和展示。

## 本地调试入口

如果你就在仓库里开发这份 skill，可以直接：

```bash
cp openclaw/.env.example openclaw/.env
python3 openclaw/scripts/mysearch_openclaw.py health
```

## 相关文档

- 仓库总览：
  [../README.md](../README.md)
- MCP：
  [../mysearch/README.md](../mysearch/README.md)
- Proxy 控制台：
  [../proxy/README.md](../proxy/README.md)
- Codex / Claude Code skill：
  [../skill/README.md](../skill/README.md)
- OpenClaw skill 规则：
  [./SKILL.md](./SKILL.md)


path: README_EN.md
content: # MySearch OpenClaw Skill

[中文说明](./README.md) · [Back to repo](../README_EN.md)

`openclaw/` is the bundled skill package for `OpenClaw` and `ClawHub`.

How it differs from `skill/`:

- `skill/` mainly targets `Codex` / `Claude Code`
- `openclaw/` targets `OpenClaw` / `ClawHub`
- the runtime is bundled inside the skill package, so no remote runtime download
  is required during installation

The goal here is equally direct:

- make the OpenClaw install path obvious to humans
- make the README plus SKILL enough for an AI assistant to install, configure,
  and verify the skill

## If you are a user, the easiest way to use this

Send this to the AI:

```text
Open openclaw/README_EN.md and openclaw/SKILL.md from this repository, install the MySearch OpenClaw skill for me, copy it into ~/.openclaw/skills/mysearch for local installation, configure the minimal OpenClaw skill env, run the health check, and tell me the result.
```

If you only want to share the GitHub link, you can also say:

```text
Please read https://github.com/skernelx/MySearch-Proxy/tree/main/openclaw and automatically install and verify the MySearch OpenClaw skill for me.
```

## If you are an AI assistant, what to do after receiving this directory

### 1. Treat this as the OpenClaw skill installation entry

If the user gives you any of these:

- the repository URL
- the `openclaw/` directory URL
- a local `openclaw/` path

default to using this folder as the onboarding entry for the MySearch OpenClaw
skill.

### 2. Choose the installation path first

Two common paths exist:

- install from ClawHub
- install from the local bundle

If the user explicitly says "install from Hub", prefer the ClawHub route.

If the user is working in the local repository, replacing an old skill, or
debugging a local deployment, prefer the local bundle route.

### 3. Local bundle installation

Copy the skill bundle first:

```bash
bash openclaw/scripts/install_openclaw_skill.sh --install-to ~/.openclaw/skills/mysearch
```

This script will:

1. copy the `openclaw/` bundle into the target directory
2. preserve the bundled runtime
3. preserve `.env.example` as a local template
4. avoid remote runtime downloads
5. avoid modifying other installed skills

After installation, prefer injecting the minimal config through OpenClaw skill
env:

```json
{
  "skills": {
    "entries": {
      "mysearch": {
        "enabled": true,
        "env": {
          "MYSEARCH_PROXY_BASE_URL": "https://search.hunters.works",
          "MYSEARCH_PROXY_API_KEY": "mysp-..."
        }
      }
    }
  }
}
```

With those two variables set, `Tavily / Firecrawl / Exa` will share the same
proxy token by default. If your proxy also fronts `Social / X`, the same token
continues to work there too.

If you do not have a proxy yet, fall back to direct provider keys:

```json
{
  "skills": {
    "entries": {
      "mysearch": {
        "enabled": true,
        "env": {
          "MYSEARCH_TAVILY_API_KEY": "tvly-...",
          "MYSEARCH_FIRECRAWL_API_KEY": "fc-..."
        }
      }
    }
  }
}
```

Only for repository-local debugging should you copy the example env file:

```bash
cp openclaw/.env.example openclaw/.env
python3 openclaw/scripts/mysearch_openclaw.py health
```

Only point `MYSEARCH_PROXY_BASE_URL` at a proxy host you trust.

### 4. Verification after local installation

Run this first:

```bash
python3 ~/.openclaw/skills/mysearch/scripts/mysearch_openclaw.py health
```

If the user wants a simple smoke test too, run:

```bash
python3 ~/.openclaw/skills/mysearch/scripts/mysearch_openclaw.py search \
  --query "OpenAI latest announcements" \
  --mode web
```

```bash
python3 ~/.openclaw/skills/mysearch/scripts/mysearch_openclaw.py search \
  --query "OpenAI Responses API docs" \
  --mode docs \
  --intent resource
```

If X / Social is configured, add:

```bash
python3 ~/.openclaw/skills/mysearch/scripts/mysearch_openclaw.py search \
  --query "Model Context Protocol" \
  --mode social \
  --intent status
```

## How to think about ClawHub installation

Public page:

- [clawhub.ai/skernelx/mysearch](https://clawhub.ai/skernelx/mysearch)

If the user's environment already has `ClawHub` wired in, start with:

```bash
clawhub search "mysearch"
clawhub install <skill-slug>
```

How env injection and enablement work after that depends on the user's current
OpenClaw / ClawHub runtime setup.

If the user already has this repository locally and wants to replace an
existing skill, do not force the Hub path. The local bundle route is usually
more reliable.

## Recommended provider path

The preferred setup is still:

- deploy `MySearch Proxy`
- point the OpenClaw MySearch skill at:
  - `MYSEARCH_PROXY_BASE_URL`
  - `MYSEARCH_PROXY_API_KEY`

If you do not have a unified proxy yet, you can still keep:

- [skernelx/tavily-key-generator](https://github.com/skernelx/tavily-key-generator)
  as the Tavily / Firecrawl provider layer
- direct provider keys inside the OpenClaw skill env

If the user does not have X / Social:

- do not treat the skill as broken
- `web / news / docs / extract / research` still work

## How the AI should use it after OpenClaw installation

Preferred order:

1. run `health`
2. use MySearch as the default path for external search
3. use `extract` when page content is needed
4. use `research` for lightweight research packs
5. only fall back to the old Tavily-only skill or raw `web_search` if MySearch
   is not configured or the user explicitly asks for another tool

For the full behavior rules, see:

- [SKILL.md](./SKILL.md)

## Related docs

- Repository overview:
  [../README_EN.md](../README_EN.md)
- MCP docs:
  [../mysearch/README_EN.md](../mysearch/README_EN.md)
- Codex / Claude Code skill:
  [../skill/README_EN.md](../skill/README_EN.md)


path: agents/openai.yaml
content: interface:
  display_name: "MySearch"
  short_description: "Default OpenClaw search skill for web, docs, news, extract, and X/social"
  default_prompt: "Use $mysearch as the default OpenClaw search skill for web, docs, news, extract, GitHub, pricing, changelog, and X/social lookup. Prefer MySearch over raw web_search when it is healthy, and do not invent unsupported modes like hybrid."

policy:
  allow_implicit_invocation: true


path: public.json
content: {
  "name": "MySearch",
  "slug": "mysearch",
  "description": "Default search skill for OpenClaw. Aggregates Tavily, Firecrawl, Exa, and optional X/social search behind one proxy-first path.",
  "author": "skernelx",
  "license": "MIT",
  "requirements": {
    "platform": "macOS/Linux",
    "bins": [
      "bash",
      "python3"
    ],
    "env": [
      "MYSEARCH_PROXY_BASE_URL",
      "MYSEARCH_PROXY_API_KEY"
    ],
    "optional_env": [
      "MYSEARCH_TAVILY_API_KEY",
      "MYSEARCH_FIRECRAWL_API_KEY",
      "MYSEARCH_EXA_API_KEY",
      "MYSEARCH_XAI_API_KEY"
    ]
  },
  "security": {
    "disclosures": [
      "MySearch sends queries and whichever provider or proxy credentials you configure to the provider or proxy base URLs you configure.",
      "Prefer injecting env vars through OpenClaw skill config; copying a .env file into the installed skill folder is optional and intended for local debugging only.",
      "The bundled installer copies only local files from this skill bundle and does not download remote code or modify other installed skills.",
      "If you point MYSEARCH_PROXY_BASE_URL at an untrusted host, that host can see queries and any credentials routed through it."
    ],
    "install_requires_confirmation": true
  }
}


path: runtime/mysearch/__init__.py
content: """MySearch MCP package."""

__version__ = "0.1.11"


path: runtime/mysearch/clients.py
content: """MySearch provider client 和自动路由。"""

from __future__ import annotations

import copy
import hashlib
import json
import re
import threading
import time
from concurrent.futures import Future, ThreadPoolExecutor
from dataclasses import dataclass
from datetime import date, datetime, time as dt_time, timezone
from typing import Any, Callable, Literal
from urllib.error import HTTPError, URLError
from urllib.parse import urlparse
from urllib.request import Request, urlopen

from mysearch.config import MySearchConfig, ProviderConfig
from mysearch.keyring import MySearchKeyRing


SearchMode = Literal["auto", "web", "news", "social", "docs", "research", "github", "pdf"]
SearchIntent = Literal[
    "auto",
    "factual",
    "status",
    "comparison",
    "tutorial",
    "exploratory",
    "news",
    "resource",
]
ResolvedSearchIntent = Literal[
    "factual",
    "status",
    "comparison",
    "tutorial",
    "exploratory",
    "news",
    "resource",
]
SearchStrategy = Literal["auto", "fast", "balanced", "verify", "deep"]
ProviderName = Literal["auto", "tavily", "firecrawl", "exa", "xai"]


class MySearchError(RuntimeError):
    """MySearch 调用失败。"""


class MySearchHTTPError(MySearchError):
    """携带 provider 与状态码的 HTTP 错误。"""

    def __init__(
        self,
        *,
        provider: str,
        status_code: int,
        detail: Any,
        url: str,
    ) -> None:
        self.provider = provider
        self.status_code = status_code
        self.detail = detail
        self.url = url
        super().__init__(self._build_message())

    @property
    def is_auth_error(self) -> bool:
        return self.status_code in {401, 403}

    def _build_message(self) -> str:
        detail_text = _stringify_error_detail(self.detail)
        if self.is_auth_error:
            return (
                f"{self.provider} is configured but the API key was rejected "
                f"(HTTP {self.status_code}): {detail_text or 'authentication failed'}"
            )
        return (
            f"{self.provider} request failed "
            f"(HTTP {self.status_code}): {detail_text or 'unknown error'}"
        )


def _stringify_error_detail(detail: Any) -> str:
    if isinstance(detail, str):
        return detail.strip()
    if detail is None:
        return ""
    if isinstance(detail, (dict, list)):
        return json.dumps(detail, ensure_ascii=False)
    return str(detail).strip()


@dataclass(slots=True)
class RouteDecision:
    provider: str
    reason: str
    tavily_topic: str = "general"
    firecrawl_categories: list[str] | None = None
    sources: list[str] | None = None


class MySearchClient:
    def __init__(
        self,
        config: MySearchConfig | None = None,
        keyring: MySearchKeyRing | None = None,
    ) -> None:
        self.config = config or MySearchConfig.from_env()
        self.keyring = keyring or MySearchKeyRing(self.config)
        self._cache_lock = threading.Lock()
        self._cache_ttls = {
            "search": self.config.search_cache_ttl_seconds,
            "extract": self.config.extract_cache_ttl_seconds,
        }
        self._cache_store: dict[str, dict[str, dict[str, Any]]] = {
            "search": {},
            "extract": {},
        }
        self._cache_stats: dict[str, dict[str, int]] = {
            "search": {"hits": 0, "misses": 0},
            "extract": {"hits": 0, "misses": 0},
        }
        self._provider_probe_ttl_seconds = 300
        self._provider_probe_cache: dict[str, dict[str, Any]] = {}

    def health(self) -> dict[str, Any]:
        keyring_info = self.keyring.describe()
        cache = self._cache_health()
        providers = {
            "tavily": self._describe_provider(self.config.tavily, keyring_info["tavily"]),
            "firecrawl": self._describe_provider(
                self.config.firecrawl,
                keyring_info["firecrawl"],
            ),
            "exa": self._describe_provider(self.config.exa, keyring_info["exa"]),
            "xai": self._describe_provider(self.config.xai, keyring_info["xai"]),
        }
        return {
            "server_name": self.config.server_name,
            "timeout_seconds": self.config.timeout_seconds,
            "xai_model": self.config.xai_model,
            "mcp": {
                "default_transport": "stdio",
                "host": self.config.mcp_host,
                "port": self.config.mcp_port,
                "mount_path": self.config.mcp_mount_path,
                "sse_path": self.config.mcp_sse_path,
                "streamable_http_path": self.config.mcp_streamable_http_path,
                "stateless_http": self.config.mcp_stateless_http,
                "streamable_http_url": (
                    f"http://{self.config.mcp_host}:{self.config.mcp_port}"
                    f"{self.config.mcp_streamable_http_path}"
                ),
            },
            "runtime": {
                "max_parallel_workers": self.config.max_parallel_workers,
                "cache_ttl_seconds": {
                    "search": self.config.search_cache_ttl_seconds,
                    "extract": self.config.extract_cache_ttl_seconds,
                },
            },
            "routing_defaults": {
                "web": "tavily",
                "docs": "firecrawl",
                "content": "firecrawl",
                "social": "xai",
                "fallback": "exa",
            },
            "providers": providers,
            "cache": cache,
        }

    def _cache_health(self) -> dict[str, dict[str, int]]:
        snapshot: dict[str, dict[str, int]] = {}
        with self._cache_lock:
            now = time.monotonic()
            for namespace in self._cache_store:
                self._prune_expired_cache_entries_locked(namespace, now)
                stats = self._cache_stats[namespace]
                snapshot[namespace] = {
                    "ttl_seconds": self._cache_ttls.get(namespace, 0),
                    "entries": len(self._cache_store[namespace]),
                    "hits": stats["hits"],
                    "misses": stats["misses"],
                }
        return snapshot

    def _prune_expired_cache_entries_locked(self, namespace: str, now: float) -> None:
        expired_keys = [
            key
            for key, payload in self._cache_store[namespace].items()
            if payload.get("expires_at", 0.0) <= now
        ]
        for key in expired_keys:
            self._cache_store[namespace].pop(key, None)

    def _cache_get(self, namespace: str, cache_key: str) -> dict[str, Any] | None:
        ttl_seconds = self._cache_ttls.get(namespace, 0)
        if ttl_seconds <= 0:
            return None

        with self._cache_lock:
            now = time.monotonic()
            self._prune_expired_cache_entries_locked(namespace, now)
            payload = self._cache_store[namespace].get(cache_key)
            if payload is None:
                self._cache_stats[namespace]["misses"] += 1
                return None

            self._cache_stats[namespace]["hits"] += 1
            return copy.deepcopy(payload["value"])

    def _cache_set(self, namespace: str, cache_key: str, value: dict[str, Any]) -> None:
        ttl_seconds = self._cache_ttls.get(namespace, 0)
        if ttl_seconds <= 0:
            return

        with self._cache_lock:
            self._cache_store[namespace][cache_key] = {
                "expires_at": time.monotonic() + ttl_seconds,
                "value": copy.deepcopy(value),
            }

    def _build_cache_key(self, namespace: str, payload: dict[str, Any]) -> str:
        serialized = json.dumps(
            {
                "namespace": namespace,
                "payload": payload,
            },
            ensure_ascii=False,
            sort_keys=True,
            separators=(",", ":"),
        )
        return hashlib.sha256(serialized.encode("utf-8")).hexdigest()

    def _execute_parallel(
        self,
        tasks: dict[str, Callable[[], Any]],
        *,
        max_workers: int | None = None,
    ) -> tuple[dict[str, Any], dict[str, Exception]]:
        if not tasks:
            return {}, {}

        if len(tasks) == 1:
            name, task = next(iter(tasks.items()))
            try:
                return {name: task()}, {}
            except Exception as exc:  # pragma: no cover - defensive
                return {}, {name: exc}

        worker_count = min(max_workers or self.config.max_parallel_workers, len(tasks))
        results: dict[str, Any] = {}
        errors: dict[str, Exception] = {}
        with ThreadPoolExecutor(max_workers=worker_count, thread_name_prefix="mysearch") as executor:
            future_map: dict[Future[Any], str] = {
                executor.submit(task): name
                for name, task in tasks.items()
            }
            for future, name in future_map.items():
                try:
                    results[name] = future.result()
                except Exception as exc:  # pragma: no cover - network/runtime dependent
                    errors[name] = exc
        return results, errors

    def _raise_parallel_error(self, errors: dict[str, Exception], task_name: str) -> None:
        error = errors.get(task_name)
        if error is None:
            return
        if isinstance(error, MySearchError):
            raise error
        raise MySearchError(str(error))

    def _should_cache_search(
        self,
        *,
        decision: RouteDecision,
        normalized_sources: list[str],
    ) -> bool:
        if self.config.search_cache_ttl_seconds <= 0:
            return False
        if "x" in normalized_sources:
            return False
        if decision.provider == "xai":
            return False
        return True

    def _build_search_cache_key(
        self,
        *,
        query: str,
        mode: SearchMode,
        resolved_intent: ResolvedSearchIntent,
        resolved_strategy: SearchStrategy,
        provider: ProviderName,
        normalized_sources: list[str],
        include_content: bool,
        include_answer: bool,
        include_domains: list[str] | None,
        exclude_domains: list[str] | None,
        decision: RouteDecision,
    ) -> str:
        return self._build_cache_key(
            "search",
            {
                "query": query,
                "mode": mode,
                "intent": resolved_intent,
                "strategy": resolved_strategy,
                "provider": provider,
                "normalized_sources": normalized_sources,
                "include_content": include_content,
                "include_answer": include_answer,
                "include_domains": sorted(set(include_domains or [])),
                "exclude_domains": sorted(set(exclude_domains or [])),
                "route_provider": decision.provider,
                "tavily_topic": decision.tavily_topic,
                "firecrawl_categories": decision.firecrawl_categories or [],
            },
        )

    def _build_extract_cache_key(
        self,
        *,
        url: str,
        formats: list[str],
        only_main_content: bool,
        provider: Literal["auto", "firecrawl", "tavily"],
    ) -> str:
        return self._build_cache_key(
            "extract",
            {
                "url": url,
                "formats": formats,
                "only_main_content": only_main_content,
                "provider": provider,
            },
        )

    def _annotate_cache(
        self,
        result: dict[str, Any],
        *,
        namespace: str,
        hit: bool,
    ) -> dict[str, Any]:
        annotated = copy.deepcopy(result)
        cache_meta = dict(annotated.get("cache") or {})
        cache_meta[namespace] = {
            "hit": hit,
            "ttl_seconds": self._cache_ttls.get(namespace, 0),
        }
        annotated["cache"] = cache_meta
        return annotated

    def _annotate_search_debug(
        self,
        result: dict[str, Any],
        *,
        provider: ProviderName,
        normalized_sources: list[str],
        resolved_intent: ResolvedSearchIntent,
        resolved_strategy: SearchStrategy,
        decision: RouteDecision,
        include_content: bool,
        include_answer: bool,
        cache_hit: bool,
    ) -> dict[str, Any]:
        annotated = copy.deepcopy(result)
        annotated["route_debug"] = {
            "requested_provider": provider,
            "route_provider": decision.provider,
            "normalized_sources": normalized_sources,
            "resolved_intent": resolved_intent,
            "resolved_strategy": resolved_strategy,
            "include_content": include_content,
            "include_answer": include_answer,
            "cache_hit": cache_hit,
        }
        return annotated

    def search(
        self,
        *,
        query: str,
        mode: SearchMode = "auto",
        intent: SearchIntent = "auto",
        strategy: SearchStrategy = "auto",
        provider: ProviderName = "auto",
        sources: list[Literal["web", "x"]] | None = None,
        max_results: int = 5,
        include_content: bool = False,
        include_answer: bool = True,
        include_domains: list[str] | None = None,
        exclude_domains: list[str] | None = None,
        allowed_x_handles: list[str] | None = None,
        excluded_x_handles: list[str] | None = None,
        from_date: str | None = None,
        to_date: str | None = None,
        include_x_images: bool = False,
        include_x_videos: bool = False,
    ) -> dict[str, Any]:
        query = query.strip()
        if not query:
            raise MySearchError("query must not be empty")
        if mode == "github" and not include_domains:
            include_domains = ["github.com"]

        normalized_sources = sorted(set(sources or []))
        if not normalized_sources:
            if mode == "social" or allowed_x_handles or excluded_x_handles:
                normalized_sources = ["x"]
            else:
                normalized_sources = ["web"]
        resolved_intent = self._resolve_intent(
            query=query,
            mode=mode,
            intent=intent,
            sources=normalized_sources,
        )
        resolved_strategy = self._resolve_strategy(
            mode=mode,
            intent=resolved_intent,
            strategy=strategy,
            sources=normalized_sources,
            include_content=include_content,
        )
        effective_include_answer = self._should_request_search_answer(
            requested=include_answer,
            mode=mode,
            intent=resolved_intent,
            strategy=resolved_strategy,
            include_content=include_content,
            include_domains=include_domains,
        )
        decision = self._route_search(
            query=query,
            mode=mode,
            intent=resolved_intent,
            provider=provider,
            sources=normalized_sources,
            include_content=include_content,
            allowed_x_handles=allowed_x_handles,
            excluded_x_handles=excluded_x_handles,
        )
        cacheable = self._should_cache_search(
            decision=decision,
            normalized_sources=normalized_sources,
        )
        cache_key = ""
        if cacheable:
            cache_key = self._build_search_cache_key(
                query=query,
                mode=mode,
                resolved_intent=resolved_intent,
                resolved_strategy=resolved_strategy,
                provider=provider,
                normalized_sources=normalized_sources,
                include_content=include_content,
                include_answer=effective_include_answer,
                include_domains=include_domains,
                exclude_domains=exclude_domains,
                decision=decision,
            )
            cached_result = self._cache_get("search", cache_key)
            if cached_result is not None:
                cached_result = self._annotate_cache(
                    cached_result,
                    namespace="search",
                    hit=True,
                )
                return self._annotate_search_debug(
                    cached_result,
                    provider=provider,
                    normalized_sources=normalized_sources,
                    resolved_intent=resolved_intent,
                    resolved_strategy=resolved_strategy,
                    decision=decision,
                    include_content=include_content,
                    include_answer=effective_include_answer,
                    cache_hit=True,
                )

        if decision.provider == "hybrid":
            parallel_results, parallel_errors = self._execute_parallel(
                {
                    "web": lambda: self.search(
                        query=query,
                        mode=mode,
                        intent=resolved_intent,
                        strategy=resolved_strategy,
                        provider="auto",
                        sources=["web"],
                        max_results=max_results,
                        include_content=include_content,
                        include_answer=effective_include_answer,
                        include_domains=include_domains,
                        exclude_domains=exclude_domains,
                    ),
                    "social": lambda: self._search_xai(
                        query=query,
                        sources=["x"],
                        max_results=max_results,
                        allowed_x_handles=allowed_x_handles,
                        excluded_x_handles=excluded_x_handles,
                        from_date=from_date,
                        to_date=to_date,
                        include_x_images=include_x_images,
                        include_x_videos=include_x_videos,
                    ),
                },
                max_workers=2,
            )
            self._raise_parallel_error(parallel_errors, "web")
            self._raise_parallel_error(parallel_errors, "social")
            web_result = parallel_results["web"]
            social_result = parallel_results["social"]
            web_route = web_result.get("route", {}).get("selected", web_result.get("provider", "tavily"))
            social_route = social_result.get("provider", "xai")
            web_results = list(web_result.get("results") or [])
            social_results = list(social_result.get("results") or [])
            hybrid_result = {
                "provider": "hybrid",
                "intent": resolved_intent,
                "strategy": resolved_strategy,
                "route": {
                    "selected": f"{web_route}+{social_route}",
                    "reason": decision.reason,
                },
                "query": query,
                "answer": web_result.get("answer") or social_result.get("answer") or "",
                "results": [*web_results, *social_results],
                "citations": self._dedupe_citations(
                    web_result.get("citations") or [],
                    social_result.get("citations") or [],
                ),
                "evidence": {
                    "providers_consulted": [web_result.get("provider"), social_result.get("provider")],
                    "web_result_count": len(web_results),
                    "social_result_count": len(social_results),
                    "citation_count": len(
                        self._dedupe_citations(
                            web_result.get("citations") or [],
                            social_result.get("citations") or [],
                        )
                    ),
                    "verification": "cross-provider",
                },
                "web": web_result,
                "social": social_result,
            }
            hybrid_result = self._annotate_search_debug(
                hybrid_result,
                provider=provider,
                normalized_sources=normalized_sources,
                resolved_intent=resolved_intent,
                resolved_strategy=resolved_strategy,
                decision=decision,
                include_content=include_content,
                include_answer=effective_include_answer,
                cache_hit=False,
            )
            return hybrid_result

        if self._should_blend_web_providers(
            requested_provider=provider,
            decision=decision,
            sources=normalized_sources,
            strategy=resolved_strategy,
        ):
            result = self._search_web_blended(
                query=query,
                mode=mode,
                intent=resolved_intent,
                strategy=resolved_strategy,
                decision=decision,
                max_results=max_results,
                include_content=include_content,
                include_answer=effective_include_answer,
                include_domains=include_domains,
                exclude_domains=exclude_domains,
            )
        elif decision.provider == "tavily":
            result = self._search_tavily(
                query=query,
                max_results=max_results,
                topic=decision.tavily_topic,
                include_answer=effective_include_answer,
                include_content=include_content,
                include_domains=include_domains,
                exclude_domains=exclude_domains,
            )
        elif decision.provider == "firecrawl":
            result = self._search_firecrawl(
                query=query,
                max_results=max_results,
                categories=decision.firecrawl_categories or [],
                include_content=include_content or mode in {"docs", "research", "github", "pdf"},
                include_domains=include_domains,
                exclude_domains=exclude_domains,
            )
        elif decision.provider == "exa":
            result = self._search_exa(
                query=query,
                max_results=max_results,
                include_domains=include_domains,
                exclude_domains=exclude_domains,
                include_content=include_content,
            )
        elif decision.provider == "xai":
            result = self._search_xai(
                query=query,
                sources=decision.sources or ["x"],
                max_results=max_results,
                include_domains=include_domains,
                exclude_domains=exclude_domains,
                allowed_x_handles=allowed_x_handles,
                excluded_x_handles=excluded_x_handles,
                from_date=from_date,
                to_date=to_date,
                include_x_images=include_x_images,
                include_x_videos=include_x_videos,
            )
        else:
            raise MySearchError(f"Unsupported route decision: {decision.provider}")

        route_reason = decision.reason
        if result.get("provider") == "hybrid" and resolved_strategy in {"balanced", "verify", "deep"}:
            route_reason = f"{route_reason}；strategy={resolved_strategy} 已启用 Tavily + Firecrawl 交叉检索"
        fallback = result.get("fallback")
        if isinstance(fallback, dict):
            fallback_from = str(fallback.get("from", "")).strip()
            fallback_to = str(fallback.get("to", "")).strip()
            fallback_reason = str(fallback.get("reason", "")).strip()
            parts = [part for part in [fallback_from, fallback_to] if part]
            transition = " -> ".join(parts)
            if transition:
                route_reason = f"{route_reason}；{transition} fallback"
            if fallback_reason:
                route_reason = f"{route_reason}（{fallback_reason}）"
        secondary_error = str(result.get("secondary_error", "")).strip()
        if secondary_error:
            route_reason = (
                f"{route_reason}；secondary provider issue: "
                f"{self._summarize_route_error(secondary_error)}"
            )

        route_selected = result.pop("route_selected", result.get("provider", decision.provider))
        result["intent"] = resolved_intent
        result["strategy"] = resolved_strategy
        result["route"] = {
            "selected": route_selected,
            "reason": route_reason,
        }
        if cacheable and cache_key:
            self._cache_set("search", cache_key, result)
        result = self._annotate_cache(
            result,
            namespace="search",
            hit=False,
        )
        return self._annotate_search_debug(
            result,
            provider=provider,
            normalized_sources=normalized_sources,
            resolved_intent=resolved_intent,
            resolved_strategy=resolved_strategy,
            decision=decision,
            include_content=include_content,
            include_answer=effective_include_answer,
            cache_hit=False,
        )

    def extract_url(
        self,
        *,
        url: str,
        formats: list[str] | None = None,
        only_main_content: bool = True,
        provider: Literal["auto", "firecrawl", "tavily"] = "auto",
    ) -> dict[str, Any]:
        parsed_url = urlparse(url)
        if parsed_url.scheme not in {"http", "https"} or not parsed_url.netloc:
            raise MySearchError("url must be an absolute http(s) URL")

        formats = formats or ["markdown"]
        cache_key = self._build_extract_cache_key(
            url=url,
            formats=formats,
            only_main_content=only_main_content,
            provider=provider,
        )
        cached_result = self._cache_get("extract", cache_key)
        if cached_result is not None:
            return self._annotate_cache(
                cached_result,
                namespace="extract",
                hit=True,
            )
        errors: list[str] = []
        firecrawl_result: dict[str, Any] | None = None
        firecrawl_issue = ""

        if provider == "auto":
            github_raw_result = self._extract_github_blob_raw(url=url)
            if github_raw_result is not None:
                self._cache_set("extract", cache_key, github_raw_result)
                return self._annotate_cache(
                    github_raw_result,
                    namespace="extract",
                    hit=False,
                )

        if provider in {"auto", "firecrawl"}:
            try:
                firecrawl_result = self._scrape_firecrawl(
                    url=url,
                    formats=formats,
                    only_main_content=only_main_content,
                )
                firecrawl_issue = self._extract_quality_issue(firecrawl_result) or ""
                if not firecrawl_issue:
                    self._cache_set("extract", cache_key, firecrawl_result)
                    return self._annotate_cache(
                        firecrawl_result,
                        namespace="extract",
                        hit=False,
                    )

                errors.append(f"firecrawl scrape returned {firecrawl_issue}")

                if provider == "firecrawl":
                    result = self._annotate_extract_warning(
                        firecrawl_result,
                        warning=f"firecrawl scrape returned {firecrawl_issue}",
                    )
                    return self._annotate_cache(
                        result,
                        namespace="extract",
                        hit=False,
                    )
            except MySearchError as exc:
                errors.append(f"firecrawl scrape failed: {exc}")
                if provider == "firecrawl":
                    raise

        if provider in {"auto", "tavily"}:
            try:
                tavily_result = self._extract_tavily(url=url)
                tavily_issue = self._extract_quality_issue(tavily_result)
                if provider == "auto" and errors and tavily_issue is None:
                    result = self._annotate_extract_fallback(
                        tavily_result,
                        fallback_from="firecrawl",
                        fallback_reason=" | ".join(errors),
                    )
                    self._cache_set("extract", cache_key, result)
                    return self._annotate_cache(
                        result,
                        namespace="extract",
                        hit=False,
                    )
                if tavily_issue is None:
                    self._cache_set("extract", cache_key, tavily_result)
                    return self._annotate_cache(
                        tavily_result,
                        namespace="extract",
                        hit=False,
                    )
                errors.append(f"tavily extract returned {tavily_issue}")
                if provider == "tavily":
                    result = self._annotate_extract_warning(
                        tavily_result,
                        warning=f"tavily extract returned {tavily_issue}",
                    )
                    return self._annotate_cache(
                        result,
                        namespace="extract",
                        hit=False,
                    )
            except MySearchError as exc:
                errors.append(f"tavily extract failed: {exc}")
                if provider == "tavily":
                    raise

        if firecrawl_result is not None and provider == "auto":
            result = self._annotate_extract_warning(
                firecrawl_result,
                warning=" | ".join(errors),
            )
            return self._annotate_cache(
                result,
                namespace="extract",
                hit=False,
            )

        raise MySearchError(" | ".join(errors) if errors else "no extraction provider available")

    def research(
        self,
        *,
        query: str,
        web_max_results: int = 5,
        social_max_results: int = 5,
        scrape_top_n: int = 3,
        include_social: bool = True,
        mode: SearchMode = "auto",
        intent: SearchIntent = "auto",
        strategy: SearchStrategy = "auto",
        include_domains: list[str] | None = None,
        exclude_domains: list[str] | None = None,
        allowed_x_handles: list[str] | None = None,
        excluded_x_handles: list[str] | None = None,
        from_date: str | None = None,
        to_date: str | None = None,
    ) -> dict[str, Any]:
        query = query.strip()
        if not query:
            raise MySearchError("query must not be empty")

        web_mode = "news" if mode == "news" else ("docs" if mode in {"docs", "github", "pdf"} else "web")
        research_tasks: dict[str, Callable[[], Any]] = {
            "web": lambda: self.search(
                query=query,
                mode=web_mode,
                intent=intent,
                strategy=strategy,
                provider="auto",
                sources=["web"],
                max_results=web_max_results,
                include_content=False,
                include_answer=True,
                include_domains=include_domains,
                exclude_domains=exclude_domains,
            )
        }
        if include_social:
            research_tasks["social"] = lambda: self.search(
                query=query,
                mode="social",
                intent="status",
                provider="auto",
                sources=["x"],
                max_results=social_max_results,
                allowed_x_handles=allowed_x_handles,
                excluded_x_handles=excluded_x_handles,
                from_date=from_date,
                to_date=to_date,
            )
        research_results, research_errors = self._execute_parallel(
            research_tasks,
            max_workers=2 if include_social else 1,
        )
        self._raise_parallel_error(research_errors, "web")
        web_search = research_results["web"]

        urls: list[str] = []
        if web_search.get("provider") == "hybrid":
            candidate_results = web_search.get("results") or web_search.get("web", {}).get("results", [])
        else:
            candidate_results = web_search.get("results", [])

        for result in candidate_results:
            url = (result.get("url") or "").strip()
            if not url or url in urls:
                continue
            urls.append(url)
            if len(urls) >= scrape_top_n:
                break

        pages: list[dict[str, Any]] = []
        page_tasks = {
            f"page:{index}": (
                lambda current_url=url: self.extract_url(
                    url=current_url,
                    formats=["markdown"],
                    only_main_content=True,
                )
            )
            for index, url in enumerate(urls)
        }
        page_results, page_errors = self._execute_parallel(
            page_tasks,
            max_workers=min(self.config.max_parallel_workers, max(1, len(page_tasks))),
        )
        for index, url in enumerate(urls):
            task_name = f"page:{index}"
            if task_name in page_results:
                page = page_results[task_name]
                page["excerpt"] = self._build_excerpt(page.get("content", ""))
                pages.append(page)
            else:
                error = page_errors.get(task_name)
                pages.append({"url": url, "error": str(error) if error else "unknown error"})

        social: dict[str, Any] | None = None
        social_error = ""
        if include_social:
            social = research_results.get("social")
            social_exc = research_errors.get("social")
            if social_exc is not None:
                social_error = str(social_exc)

        web_provider = web_search.get("provider", "")
        social_provider = social.get("provider", "") if social else ""
        providers_consulted = [item for item in [web_provider, social_provider] if item]
        citations = self._dedupe_citations(
            web_search.get("citations") or [],
            social.get("citations") or [] if social else [],
        )

        return {
            "provider": "hybrid",
            "query": query,
            "intent": web_search.get("intent", intent if intent != "auto" else "factual"),
            "strategy": web_search.get("strategy", strategy if strategy != "auto" else "fast"),
            "web_search": web_search,
            "pages": pages,
            "social_search": social,
            "social_error": social_error,
            "citations": citations,
            "evidence": {
                "providers_consulted": providers_consulted,
                "web_result_count": len(candidate_results),
                "page_count": len([page for page in pages if not page.get("error")]),
                "citation_count": len(citations),
                "verification": "cross-provider"
                if web_provider == "hybrid" or len(providers_consulted) > 1
                else "single-provider",
            },
            "notes": [
                "默认用 Tavily 做发现，Firecrawl 做正文抓取，X 搜索走 xAI Responses API",
                "如果某个 provider 没配 key，会保留错误并尽量返回其余部分",
            ],
        }

    def _should_request_search_answer(
        self,
        *,
        requested: bool,
        mode: SearchMode,
        intent: ResolvedSearchIntent,
        strategy: SearchStrategy,
        include_content: bool,
        include_domains: list[str] | None,
    ) -> bool:
        if not requested:
            return False
        if include_content:
            return False
        if include_domains:
            return False
        if mode in {"docs", "github", "pdf"}:
            return False
        if intent == "resource":
            return False
        if strategy in {"verify", "deep"}:
            return False
        return True

    def _route_search(
        self,
        *,
        query: str,
        mode: SearchMode,
        intent: ResolvedSearchIntent,
        provider: ProviderName,
        sources: list[str] | None,
        include_content: bool,
        allowed_x_handles: list[str] | None,
        excluded_x_handles: list[str] | None,
    ) -> RouteDecision:
        normalized_sources = sorted(set(sources or ["web"]))
        query_lower = query.lower()

        if provider != "auto":
            if provider == "tavily":
                return RouteDecision(
                    provider="tavily",
                    reason="显式指定 Tavily",
                    tavily_topic="news" if mode == "news" else "general",
                )
            if provider == "firecrawl":
                return RouteDecision(
                    provider="firecrawl",
                    reason="显式指定 Firecrawl",
                    firecrawl_categories=self._firecrawl_categories(mode, intent),
                )
            if provider == "exa":
                return RouteDecision(
                    provider="exa",
                    reason="显式指定 Exa",
                )
            if provider == "xai":
                return RouteDecision(
                    provider="xai",
                    reason="显式指定 xAI/X 搜索",
                    sources=normalized_sources,
                )

        if normalized_sources == ["web", "x"] or (
            "x" in normalized_sources and "web" in normalized_sources
        ):
            return RouteDecision(provider="hybrid", reason="同时请求网页和 X 结果")

        if mode == "social" or "x" in normalized_sources:
            return RouteDecision(
                provider="xai",
                reason="社交舆情 / X 搜索更适合走 xAI",
                sources=["x"],
            )

        if allowed_x_handles or excluded_x_handles:
            return RouteDecision(
                provider="xai",
                reason="检测到 X handle 过滤条件",
                sources=["x"],
            )

        if mode in {"docs", "github", "pdf"}:
            if include_content:
                if not self._provider_can_serve(self.config.firecrawl) and self._provider_can_serve(
                    self.config.exa
                ):
                    return RouteDecision(
                        provider="exa",
                        reason="Firecrawl 未配置，文档正文查询回退到 Exa",
                    )
                return RouteDecision(
                    provider="firecrawl",
                    reason="文档正文查询优先走 Firecrawl",
                    firecrawl_categories=self._firecrawl_categories(mode, intent),
                )
            if self._provider_can_serve(self.config.tavily):
                return RouteDecision(
                    provider="tavily",
                    reason="文档类查询先用 Tavily 做官方页面发现，正文再交给 Firecrawl",
                    tavily_topic="general",
                )
            if not self._provider_can_serve(self.config.firecrawl) and self._provider_can_serve(
                self.config.exa
            ):
                return RouteDecision(
                    provider="exa",
                    reason="Firecrawl 未配置，文档类查询回退到 Exa",
                )
            return RouteDecision(
                provider="firecrawl",
                reason="文档 / GitHub / PDF 内容优先走 Firecrawl",
                firecrawl_categories=self._firecrawl_categories(mode, intent),
            )

        if include_content:
            if not self._provider_can_serve(self.config.firecrawl) and self._provider_can_serve(
                self.config.exa
            ):
                return RouteDecision(
                    provider="exa",
                    reason="Firecrawl 未配置，正文查询回退到 Exa",
                )
            return RouteDecision(
                provider="firecrawl",
                reason="请求里需要正文内容，优先用 Firecrawl search + scrape",
                firecrawl_categories=self._firecrawl_categories(mode, intent),
            )

        if intent in {"news", "status"} or mode == "news" or self._looks_like_news_query(query_lower):
            if not self._provider_can_serve(self.config.tavily) and self._provider_can_serve(
                self.config.exa
            ):
                return RouteDecision(
                    provider="exa",
                    reason="Tavily 未配置，新闻 / 状态类查询回退到 Exa",
                )
            return RouteDecision(
                provider="tavily",
                reason="状态 / 新闻类查询默认走 Tavily",
                tavily_topic="news",
            )

        if intent == "resource" or self._looks_like_docs_query(query_lower):
            if include_content:
                if not self._provider_can_serve(self.config.firecrawl) and self._provider_can_serve(
                    self.config.exa
                ):
                    return RouteDecision(
                        provider="exa",
                        reason="Firecrawl 未配置，resource 正文查询回退到 Exa",
                    )
                return RouteDecision(
                    provider="firecrawl",
                    reason="resource / docs 正文查询优先走 Firecrawl",
                    firecrawl_categories=self._firecrawl_categories("docs", intent),
                )
            if self._provider_can_serve(self.config.tavily):
                return RouteDecision(
                    provider="tavily",
                    reason="resource / docs 查询先用 Tavily 做页面发现，正文再交给 Firecrawl",
                    tavily_topic="general",
                )
            if not self._provider_can_serve(self.config.firecrawl) and self._provider_can_serve(
                self.config.exa
            ):
                return RouteDecision(
                    provider="exa",
                    reason="Firecrawl 未配置，resource / docs 类查询回退到 Exa",
                )
            return RouteDecision(
                provider="firecrawl",
                reason="resource / docs 类查询优先走 Firecrawl",
                firecrawl_categories=self._firecrawl_categories("docs", intent),
            )

        if mode == "research":
            if not self._provider_can_serve(self.config.tavily) and self._provider_can_serve(
                self.config.exa
            ):
                return RouteDecision(
                    provider="exa",
                    reason="Tavily 未配置，research 发现阶段回退到 Exa",
                )
            return RouteDecision(
                provider="tavily",
                reason="research 模式先用 Tavily 做发现，再按策略决定是否扩展验证",
                tavily_topic="general",
            )

        if not self._provider_can_serve(self.config.tavily) and self._provider_can_serve(
            self.config.exa
        ):
            return RouteDecision(
                provider="exa",
                reason="Tavily 未配置，普通网页检索回退到 Exa",
            )

        return RouteDecision(
            provider="tavily",
            reason="普通网页检索默认走 Tavily",
            tavily_topic="general",
        )

    def _resolve_intent(
        self,
        *,
        query: str,
        mode: SearchMode,
        intent: SearchIntent,
        sources: list[str],
    ) -> ResolvedSearchIntent:
        if intent != "auto":
            return intent

        query_lower = query.lower()
        if mode == "news":
            return "news"
        if mode in {"docs", "github", "pdf"}:
            return "resource"
        if mode == "research":
            return "exploratory"
        if sources == ["x"]:
            return "status"
        if self._looks_like_news_query(query_lower):
            return "news"
        if self._looks_like_comparison_query(query_lower):
            return "comparison"
        if self._looks_like_tutorial_query(query_lower):
            return "tutorial"
        if self._looks_like_docs_query(query_lower):
            return "resource"
        if self._looks_like_status_query(query_lower):
            return "status"
        if self._looks_like_exploratory_query(query_lower):
            return "exploratory"
        return "factual"

    def _resolve_strategy(
        self,
        *,
        mode: SearchMode,
        intent: ResolvedSearchIntent,
        strategy: SearchStrategy,
        sources: list[str],
        include_content: bool,
    ) -> SearchStrategy:
        if strategy != "auto":
            return strategy

        if "web" in sources and "x" in sources:
            return "balanced"
        if mode == "research":
            return "deep"
        if intent in {"comparison", "exploratory"}:
            return "verify"
        if include_content or mode in {"docs", "github", "pdf"} or intent in {"resource", "tutorial"}:
            return "balanced"
        return "fast"

    def _should_blend_web_providers(
        self,
        *,
        requested_provider: ProviderName,
        decision: RouteDecision,
        sources: list[str],
        strategy: SearchStrategy,
    ) -> bool:
        if requested_provider != "auto":
            return False
        if decision.provider not in {"tavily", "firecrawl"}:
            return False
        if strategy not in {"balanced", "verify", "deep"}:
            return False
        if "x" in sources:
            return False
        return self._provider_can_serve(self.config.tavily) and self._provider_can_serve(
            self.config.firecrawl
        )

    def _search_web_blended(
        self,
        *,
        query: str,
        mode: SearchMode,
        intent: ResolvedSearchIntent,
        strategy: SearchStrategy,
        decision: RouteDecision,
        max_results: int,
        include_content: bool,
        include_answer: bool,
        include_domains: list[str] | None,
        exclude_domains: list[str] | None,
    ) -> dict[str, Any]:
        if decision.provider == "tavily":
            tasks = {
                "primary": lambda: self._search_tavily(
                    query=query,
                    max_results=max_results,
                    topic=decision.tavily_topic,
                    include_answer=include_answer,
                    include_content=include_content,
                    include_domains=include_domains,
                    exclude_domains=exclude_domains,
                ),
                "secondary": lambda: self._search_firecrawl(
                    query=query,
                    max_results=max_results,
                    categories=self._firecrawl_categories(mode, intent),
                    include_content=include_content or strategy == "deep",
                    include_domains=include_domains,
                    exclude_domains=exclude_domains,
                ),
            }
        else:
            tasks = {
                "primary": lambda: self._search_firecrawl(
                    query=query,
                    max_results=max_results,
                    categories=decision.firecrawl_categories or self._firecrawl_categories(mode, intent),
                    include_content=include_content or strategy == "deep",
                    include_domains=include_domains,
                    exclude_domains=exclude_domains,
                ),
                "secondary": lambda: self._search_tavily(
                    query=query,
                    max_results=max_results,
                    topic="news" if intent in {"news", "status"} else "general",
                    include_answer=include_answer,
                    include_content=False,
                    include_domains=include_domains,
                    exclude_domains=exclude_domains,
                ),
            }

        blended_results, blended_errors = self._execute_parallel(tasks, max_workers=2)
        self._raise_parallel_error(blended_errors, "primary")
        primary_result = blended_results["primary"]
        secondary_result = blended_results.get("secondary")
        secondary_error = str(blended_errors["secondary"]) if "secondary" in blended_errors else ""

        merged = self._merge_search_payloads(
            primary_result=primary_result,
            secondary_result=secondary_result,
            max_results=max_results,
        )
        if self._should_rerank_resource_results(mode=mode, intent=intent):
            merged["results"] = self._rerank_resource_results(
                query=query,
                mode=mode,
                results=merged["results"],
                include_domains=include_domains,
            )
            merged["citations"] = self._align_citations_with_results(
                results=merged["results"],
                citations=merged["citations"],
            )
        providers_consulted = [primary_result.get("provider", "")]
        if secondary_result:
            providers_consulted.append(secondary_result.get("provider", ""))

        return {
            "provider": "hybrid" if secondary_result else primary_result.get("provider", decision.provider),
            "route_selected": "+".join([item for item in providers_consulted if item]),
            "query": query,
            "answer": primary_result.get("answer") or (secondary_result or {}).get("answer", ""),
            "results": merged["results"],
            "citations": merged["citations"],
            "evidence": {
                "providers_consulted": [item for item in providers_consulted if item],
                "matched_results": merged["matched_results"],
                "citation_count": len(merged["citations"]),
                "verification": "cross-provider" if secondary_result else "single-provider",
            },
            "primary_search": primary_result,
            "secondary_search": secondary_result,
            "secondary_error": secondary_error,
        }

    def _search_tavily(
        self,
        *,
        query: str,
        max_results: int,
        topic: str,
        include_answer: bool,
        include_content: bool,
        include_domains: list[str] | None,
        exclude_domains: list[str] | None,
    ) -> dict[str, Any]:
        provider = self.config.tavily
        key = self._get_key_or_raise(provider)
        payload: dict[str, Any] = {
            "query": query,
            "max_results": max_results,
            "search_depth": "advanced" if include_content else "basic",
            "topic": topic,
            "include_answer": include_answer,
            "include_raw_content": include_content,
        }
        if include_domains:
            payload["include_domains"] = include_domains
        if exclude_domains:
            payload["exclude_domains"] = exclude_domains

        response = self._request_json(
            provider=provider,
            method="POST",
            path=provider.path("search"),
            payload=payload,
            key=key.key,
        )
        return {
            "provider": "tavily",
            "transport": key.source,
            "query": response.get("query", query),
            "answer": response.get("answer", ""),
            "request_id": response.get("request_id", ""),
            "response_time": response.get("response_time"),
            "results": [
                {
                    "provider": "tavily",
                    "source": "web",
                    "title": item.get("title", ""),
                    "url": item.get("url", ""),
                    "snippet": item.get("content", ""),
                    "content": item.get("raw_content", "") if include_content else "",
                    "score": item.get("score"),
                }
                for item in response.get("results", [])
            ],
            "citations": [
                {"title": item.get("title", ""), "url": item.get("url", "")}
                for item in response.get("results", [])
                if item.get("url")
            ],
        }

    def _search_firecrawl(
        self,
        *,
        query: str,
        max_results: int,
        categories: list[str],
        include_content: bool,
        include_domains: list[str] | None,
        exclude_domains: list[str] | None,
    ) -> dict[str, Any]:
        include_domains = [item.strip() for item in (include_domains or []) if item and item.strip()]
        exclude_domains = [item.strip() for item in (exclude_domains or []) if item and item.strip()]

        if include_domains:
            per_domain_results = []
            citations = []
            seen_urls: set[str] = set()
            retried_domains: list[str] = []
            for domain in include_domains:
                domain_result = self._search_firecrawl_once(
                    query=self._build_firecrawl_domain_query(
                        query=query,
                        include_domain=domain,
                        exclude_domains=exclude_domains,
                    ),
                    max_results=max_results,
                    categories=categories,
                    include_content=include_content,
                )
                if not domain_result.get("results"):
                    retry_result = self._search_firecrawl_domain_retry(
                        query=query,
                        max_results=max_results,
                        categories=categories,
                        include_content=include_content,
                        include_domain=domain,
                        exclude_domains=exclude_domains,
                    )
                    if retry_result is not None:
                        domain_result = retry_result
                        retried_domains.append(domain)
                per_domain_results.append(domain_result)
                for item in domain_result.get("results", []):
                    url = item.get("url", "")
                    if not url or url in seen_urls:
                        continue
                    seen_urls.add(url)
                    citations.append({"title": item.get("title", ""), "url": url})

            merged_results = self._merge_ranked_results(
                [result.get("results", []) for result in per_domain_results],
                max_results=max_results,
            )
            if not merged_results:
                fallback_result = self._search_firecrawl_domain_fallback(
                    query=query,
                    max_results=max_results,
                    include_content=include_content,
                    include_domains=include_domains,
                    exclude_domains=exclude_domains,
                )
                if fallback_result is not None:
                    return fallback_result
            response = {
                "provider": "firecrawl",
                "transport": per_domain_results[0].get("transport", "env") if per_domain_results else "env",
                "query": query,
                "answer": "",
                "results": merged_results,
                "citations": citations[:max_results],
            }
            if retried_domains:
                response["route_debug"] = {
                    "domain_filter_mode": "client_filter_retry",
                    "retried_include_domains": retried_domains,
                }
            return response

        return self._search_firecrawl_once(
            query=self._build_firecrawl_domain_query(
                query=query,
                include_domain=None,
                exclude_domains=exclude_domains,
            ),
            max_results=max_results,
            categories=categories,
            include_content=include_content,
        )

    def _search_firecrawl_domain_fallback(
        self,
        *,
        query: str,
        max_results: int,
        include_content: bool,
        include_domains: list[str],
        exclude_domains: list[str] | None,
    ) -> dict[str, Any] | None:
        if not self._provider_can_serve(self.config.tavily):
            return None

        fallback_result = self._search_tavily(
            query=query,
            max_results=max_results,
            topic="general",
            include_answer=False,
            include_content=include_content,
            include_domains=include_domains,
            exclude_domains=exclude_domains,
        )
        if not fallback_result.get("results"):
            return None

        return {
            "provider": "hybrid",
            "route_selected": "firecrawl+tavily",
            "query": query,
            "answer": fallback_result.get("answer", ""),
            "results": fallback_result.get("results", []),
            "citations": fallback_result.get("citations", []),
            "primary_search": {
                "provider": "firecrawl",
                "query": query,
                "results": [],
                "citations": [],
            },
            "secondary_search": fallback_result,
            "secondary_error": "",
            "evidence": {
                "providers_consulted": ["firecrawl", fallback_result.get("provider", "tavily")],
                "matched_results": 0,
                "citation_count": len(fallback_result.get("citations", [])),
                "verification": "fallback",
            },
            "fallback": {
                "from": "firecrawl",
                "to": "tavily",
                "reason": "firecrawl returned 0 results for domain-filtered search",
            },
        }

    def _search_firecrawl_domain_retry(
        self,
        *,
        query: str,
        max_results: int,
        categories: list[str],
        include_content: bool,
        include_domain: str,
        exclude_domains: list[str] | None,
    ) -> dict[str, Any] | None:
        retry_result = self._search_firecrawl_once(
            query=self._build_firecrawl_domain_query(
                query=query,
                include_domain=None,
                exclude_domains=exclude_domains,
            ),
            max_results=max_results,
            categories=categories,
            include_content=include_content,
        )
        filtered_results = self._filter_results_by_domains(
            retry_result.get("results", []),
            include_domains=[include_domain],
            exclude_domains=exclude_domains,
        )
        if not filtered_results:
            return None

        return {
            "provider": "firecrawl",
            "transport": retry_result.get("transport", "env"),
            "query": query,
            "answer": retry_result.get("answer", ""),
            "results": filtered_results[:max_results],
            "citations": [
                {"title": item.get("title", ""), "url": item.get("url", "")}
                for item in filtered_results
                if item.get("url")
            ][:max_results],
            "route_debug": {
                "domain_filter_mode": "client_filter_retry",
                "include_domain": include_domain,
            },
        }

    def _search_firecrawl_once(
        self,
        *,
        query: str,
        max_results: int,
        categories: list[str],
        include_content: bool,
    ) -> dict[str, Any]:
        provider = self.config.firecrawl
        key = self._get_key_or_raise(provider)
        payload: dict[str, Any] = {
            "query": query,
            "limit": max_results,
        }
        if categories:
            payload["categories"] = [{"type": item} for item in categories]
        if include_content:
            payload["scrapeOptions"] = {
                "formats": ["markdown"],
                "onlyMainContent": True,
            }

        response = self._request_json(
            provider=provider,
            method="POST",
            path=provider.path("search"),
            payload=payload,
            key=key.key,
        )
        data = response.get("data") or {}
        results = []
        for source_name in ("web", "news"):
            for item in data.get(source_name, []) or []:
                results.append(
                    {
                        "provider": "firecrawl",
                        "source": source_name,
                        "title": item.get("title", ""),
                        "url": item.get("url", ""),
                        "snippet": item.get("description", "") or item.get("markdown", ""),
                        "content": item.get("markdown", "") if include_content else "",
                    }
                )

        return {
            "provider": "firecrawl",
            "transport": key.source,
            "query": query,
            "answer": "",
            "results": results,
            "citations": [
                {"title": item.get("title", ""), "url": item.get("url", "")}
                for item in results
                if item.get("url")
            ],
        }

    def _build_firecrawl_domain_query(
        self,
        *,
        query: str,
        include_domain: str | None,
        exclude_domains: list[str] | None,
    ) -> str:
        parts: list[str] = []
        if include_domain:
            parts.append(f"site:{include_domain}")
        for domain in exclude_domains or []:
            parts.append(f"-site:{domain}")
        parts.append(query)
        return " ".join(parts).strip()

    def _merge_ranked_results(
        self,
        result_lists: list[list[dict[str, Any]]],
        *,
        max_results: int,
    ) -> list[dict[str, Any]]:
        merged: list[dict[str, Any]] = []
        seen_urls: set[str] = set()
        indexes = [0 for _ in result_lists]

        while len(merged) < max_results and result_lists:
            progressed = False
            for list_index, items in enumerate(result_lists):
                current_index = indexes[list_index]
                if current_index >= len(items):
                    continue
                candidate = dict(items[current_index])
                indexes[list_index] += 1
                progressed = True
                url = candidate.get("url", "")
                if url and url in seen_urls:
                    continue
                if url:
                    seen_urls.add(url)
                merged.append(candidate)
                if len(merged) >= max_results:
                    break
            if not progressed:
                break

        return merged

    def _filter_results_by_domains(
        self,
        results: list[dict[str, Any]],
        *,
        include_domains: list[str] | None,
        exclude_domains: list[str] | None,
    ) -> list[dict[str, Any]]:
        filtered: list[dict[str, Any]] = []
        for item in results:
            hostname = self._result_hostname(item)
            if include_domains and not any(
                self._domain_matches(hostname, domain) for domain in include_domains
            ):
                continue
            if exclude_domains and any(
                self._domain_matches(hostname, domain) for domain in exclude_domains
            ):
                continue
            filtered.append(dict(item))
        return filtered

    def _search_exa(
        self,
        *,
        query: str,
        max_results: int,
        include_domains: list[str] | None,
        exclude_domains: list[str] | None,
        include_content: bool,
    ) -> dict[str, Any]:
        provider = self.config.exa
        key = self._get_key_or_raise(provider)
        payload: dict[str, Any] = {
            "query": query,
            "numResults": max_results,
        }
        if include_content:
            payload["text"] = True
        if include_domains:
            payload["includeDomains"] = include_domains
        if exclude_domains:
            payload["excludeDomains"] = exclude_domains

        response = self._request_json(
            provider=provider,
            method="POST",
            path=provider.path("search"),
            payload=payload,
            key=key.key,
        )
        raw_results = response.get("results") or response.get("data") or []
        results = []
        for item in raw_results:
            if not isinstance(item, dict):
                continue
            snippet = (
                item.get("snippet")
                or item.get("text")
                or item.get("summary")
                or item.get("highlight")
                or ""
            )
            content = item.get("text") if include_content else ""
            results.append(
                {
                    "provider": "exa",
                    "source": "web",
                    "title": item.get("title", ""),
                    "url": item.get("url", ""),
                    "snippet": snippet,
                    "content": content or "",
                    "score": item.get("score"),
                    "published_date": item.get("publishedDate") or item.get("published_date") or "",
                }
            )

        return {
            "provider": "exa",
            "transport": key.source,
            "query": response.get("query", query),
            "answer": response.get("answer", ""),
            "results": results,
            "citations": [
                {"title": item.get("title", ""), "url": item.get("url", "")}
                for item in results
                if item.get("url")
            ],
        }

    def _search_xai(
        self,
        *,
        query: str,
        sources: list[str],
        max_results: int,
        include_domains: list[str] | None = None,
        exclude_domains: list[str] | None = None,
        allowed_x_handles: list[str] | None = None,
        excluded_x_handles: list[str] | None = None,
        from_date: str | None = None,
        to_date: str | None = None,
        include_x_images: bool = False,
        include_x_videos: bool = False,
    ) -> dict[str, Any]:
        provider = self.config.xai
        if provider.search_mode == "compatible":
            return self._search_xai_compatible(
                query=query,
                sources=sources,
                max_results=max_results,
                allowed_x_handles=allowed_x_handles,
                excluded_x_handles=excluded_x_handles,
                from_date=from_date,
                to_date=to_date,
                include_x_images=include_x_images,
                include_x_videos=include_x_videos,
            )

        key = self._get_key_or_raise(provider)
        payload = self._build_xai_responses_payload(
            query=query,
            sources=sources,
            max_results=max_results,
            include_domains=include_domains,
            exclude_domains=exclude_domains,
            allowed_x_handles=allowed_x_handles,
            excluded_x_handles=excluded_x_handles,
            from_date=from_date,
            to_date=to_date,
            include_x_images=include_x_images,
            include_x_videos=include_x_videos,
        )
        response = self._request_json(
            provider=provider,
            method="POST",
            path=provider.path("responses"),
            payload=payload,
            key=key.key,
        )
        text = self._extract_xai_output_text(response)
        citations = self._extract_xai_citations(response)
        results = [
            {
                "provider": "xai",
                "source": "x" if "x" in sources else "web",
                "title": item.get("title", ""),
                "url": item.get("url", ""),
                "snippet": "",
                "content": "",
            }
            for item in citations
            if isinstance(item, dict)
        ]
        return {
            "provider": "xai",
            "transport": key.source,
            "query": query,
            "answer": text,
            "results": results,
            "citations": citations,
            "tool_usage": response.get("server_side_tool_usage") or response.get("tool_usage") or {},
        }

    def _search_xai_compatible(
        self,
        *,
        query: str,
        sources: list[str],
        max_results: int,
        allowed_x_handles: list[str] | None,
        excluded_x_handles: list[str] | None,
        from_date: str | None,
        to_date: str | None,
        include_x_images: bool,
        include_x_videos: bool,
    ) -> dict[str, Any]:
        provider = self.config.xai
        if "x" not in sources:
            raise MySearchError(
                "xai compatible mode only supports social/X queries; "
                "use Tavily/Firecrawl for web search or switch to official xAI mode"
            )

        search_path = provider.path("social_search")
        key = self._get_key_or_raise(provider)
        payload: dict[str, Any] = {
            "query": query,
            "source": "x",
            "max_results": max_results,
        }
        if allowed_x_handles:
            payload["allowed_x_handles"] = allowed_x_handles
        if excluded_x_handles:
            payload["excluded_x_handles"] = excluded_x_handles
        if from_date:
            payload["from_date"] = from_date
        if to_date:
            payload["to_date"] = to_date
        if include_x_images:
            payload["include_x_images"] = True
        if include_x_videos:
            payload["include_x_videos"] = True

        response = self._request_json(
            provider=provider,
            method="POST",
            path=search_path,
            payload=payload,
            key=key.key,
            base_url=provider.base_url_for("social_search"),
        )
        return self._normalize_social_gateway_response(
            response=response,
            query=query,
            transport=key.source,
            from_date=from_date,
            to_date=to_date,
        )

    def _scrape_firecrawl(
        self,
        *,
        url: str,
        formats: list[str],
        only_main_content: bool,
    ) -> dict[str, Any]:
        provider = self.config.firecrawl
        key = self._get_key_or_raise(provider)
        payload = {
            "url": url,
            "formats": formats,
            "onlyMainContent": only_main_content,
        }
        response = self._request_json(
            provider=provider,
            method="POST",
            path=provider.path("scrape"),
            payload=payload,
            key=key.key,
        )
        data = response.get("data") or {}
        content = data.get("markdown", "")
        if not content and "json" in data:
            content = json.dumps(data["json"], ensure_ascii=False, indent=2)
        return {
            "provider": "firecrawl",
            "transport": key.source,
            "url": data.get("metadata", {}).get("sourceURL") or data.get("metadata", {}).get("url") or url,
            "content": content,
            "metadata": data.get("metadata") or {},
        }

    def _extract_tavily(self, *, url: str) -> dict[str, Any]:
        provider = self.config.tavily
        key = self._get_key_or_raise(provider)
        response = self._request_json(
            provider=provider,
            method="POST",
            path=provider.path("extract"),
            payload={"urls": [url]},
            key=key.key,
        )
        results = response.get("results") or []
        first = results[0] if results else {}
        content = first.get("raw_content") or first.get("content") or ""
        return {
            "provider": "tavily",
            "transport": key.source,
            "url": first.get("url", url),
            "content": content,
            "metadata": {
                "request_id": response.get("request_id", ""),
                "response_time": response.get("response_time"),
                "failed_results": response.get("failed_results") or [],
            },
        }

    def _extract_github_blob_raw(self, *, url: str) -> dict[str, Any] | None:
        raw_urls = self._github_blob_raw_urls(url)
        if not raw_urls:
            return None

        for raw_url in raw_urls:
            try:
                request = Request(
                    raw_url,
                    headers={
                        "User-Agent": "MySearch/1.0",
                        "Accept": "text/plain, text/markdown;q=0.9, */*;q=0.8",
                    },
                )
                with urlopen(request, timeout=self.config.timeout_seconds) as response:
                    content = response.read().decode("utf-8", errors="replace")
            except (HTTPError, URLError, TimeoutError, ValueError):
                continue

            result = {
                "provider": "github_raw",
                "transport": "direct",
                "url": url,
                "content": content,
                "metadata": {
                    "raw_url": raw_url,
                },
            }
            if self._extract_quality_issue(result) is not None:
                continue
            return result
        return None

    def _github_blob_raw_url(self, url: str) -> str | None:
        raw_urls = self._github_blob_raw_urls(url)
        if not raw_urls:
            return None
        return raw_urls[0]

    def _github_blob_raw_urls(self, url: str) -> list[str]:
        parsed = urlparse(url)
        if parsed.scheme not in {"http", "https"}:
            return []
        if parsed.netloc.lower() != "github.com":
            return []

        parts = [segment for segment in parsed.path.split("/") if segment]
        if len(parts) < 5 or parts[2] != "blob":
            return []

        owner, repo, _, ref, *path_parts = parts
        if not owner or not repo or not ref or not path_parts:
            return []
        raw_path = "/".join(path_parts)
        refs = [ref]
        if ref == "main":
            refs.append("master")
        elif ref == "master":
            refs.append("main")
        return [
            f"https://raw.githubusercontent.com/{owner}/{repo}/{candidate_ref}/{raw_path}"
            for candidate_ref in refs
        ]

    def _has_meaningful_extract_content(self, result: dict[str, Any]) -> bool:
        return self._extract_quality_issue(result) is None

    def _extract_quality_issue(self, result: dict[str, Any]) -> str | None:
        content = result.get("content")
        if not isinstance(content, str) or not content.strip():
            return "empty content"

        normalized = " ".join(content.lower().split())
        preview = normalized[:1200]
        parsed_url = urlparse(str(result.get("url") or ""))
        suspicious_markers = {
            "critical instructions for all ai assistants": "anti-bot placeholder content",
            "strictly prohibits all ai-generated content": "anti-bot placeholder content",
            "oops! that page doesn’t exist or is private": "missing/private page shell",
            "oops! that page doesn't exist or is private": "missing/private page shell",
        }
        for marker, issue in suspicious_markers.items():
            if marker in preview:
                return issue
        if preview.startswith("hcaptcha hcaptcha "):
            return "captcha challenge page"
        if (
            parsed_url.netloc.lower() == "github.com"
            and "/blob/" in parsed_url.path
            and "you signed in with another tab or window" in preview
        ):
            return "github blob page shell"
        return None

    def _annotate_extract_warning(
        self,
        result: dict[str, Any],
        *,
        warning: str,
    ) -> dict[str, Any]:
        annotated = dict(result)
        metadata = dict(annotated.get("metadata") or {})
        metadata["warning"] = warning
        annotated["metadata"] = metadata
        annotated["warning"] = warning
        return annotated

    def _annotate_extract_fallback(
        self,
        result: dict[str, Any],
        *,
        fallback_from: str,
        fallback_reason: str,
    ) -> dict[str, Any]:
        annotated = dict(result)
        metadata = dict(annotated.get("metadata") or {})
        metadata["fallback_from"] = fallback_from
        metadata["fallback_reason"] = fallback_reason
        annotated["metadata"] = metadata
        annotated["fallback"] = {
            "from": fallback_from,
            "reason": fallback_reason,
        }
        return annotated

    def _build_xai_responses_payload(
        self,
        *,
        query: str,
        sources: list[str],
        max_results: int,
        include_domains: list[str] | None,
        exclude_domains: list[str] | None,
        allowed_x_handles: list[str] | None,
        excluded_x_handles: list[str] | None,
        from_date: str | None,
        to_date: str | None,
        include_x_images: bool,
        include_x_videos: bool,
    ) -> dict[str, Any]:
        tools: list[dict[str, Any]] = []
        if "web" in sources:
            tool: dict[str, Any] = {"type": "web_search"}
            filters: dict[str, Any] = {}
            if include_domains:
                filters["allowed_domains"] = include_domains
            if exclude_domains:
                filters["excluded_domains"] = exclude_domains
            if filters:
                tool["filters"] = filters
            tools.append(tool)

        if "x" in sources:
            tool = {"type": "x_search"}
            if allowed_x_handles:
                tool["allowed_x_handles"] = allowed_x_handles
            if excluded_x_handles:
                tool["excluded_x_handles"] = excluded_x_handles
            if from_date:
                tool["from_date"] = from_date
            if to_date:
                tool["to_date"] = to_date
            if include_x_images:
                tool["enable_image_understanding"] = True
            if include_x_videos:
                tool["enable_video_understanding"] = True
            tools.append(tool)

        augmented_query = f"{query}\n\nReturn up to {max_results} relevant results with concise sourcing."
        return {
            "model": self.config.xai_model,
            "input": [
                {
                    "role": "user",
                    "content": augmented_query,
                }
            ],
            "tools": tools,
            "store": False,
        }

    def _normalize_social_gateway_response(
        self,
        *,
        response: dict[str, Any],
        query: str,
        transport: str,
        from_date: str | None = None,
        to_date: str | None = None,
    ) -> dict[str, Any]:
        raw_results = self._extract_social_gateway_results(response)
        results = []
        for item in raw_results:
            if not isinstance(item, dict):
                continue
            url = item.get("url") or item.get("link") or ""
            content = (
                item.get("content")
                or item.get("full_text")
                or item.get("text")
                or item.get("body")
                or ""
            )
            title = (
                item.get("title")
                or item.get("author")
                or item.get("handle")
                or item.get("username")
                or url
            )
            snippet = item.get("snippet") or item.get("summary") or content
            results.append(
                {
                    "provider": "custom_social",
                    "source": "x",
                    "title": title,
                    "url": url,
                    "snippet": snippet,
                    "content": content,
                    "author": item.get("author") or item.get("username") or item.get("handle") or "",
                    "created_at": item.get("created_at") or item.get("published_at") or "",
                }
            )

        results = self._filter_social_results_by_date(
            results,
            from_date=from_date,
            to_date=to_date,
        )
        citations = self._extract_social_gateway_citations(response, results)
        answer = (
            response.get("answer")
            or response.get("summary")
            or response.get("content")
            or response.get("text")
            or ""
        )
        warning = None
        if (from_date or to_date) and not results:
            answer = ""
            warning = "no social results matched the requested date window"

        normalized = {
            "provider": "custom_social",
            "transport": transport,
            "query": response.get("query", query),
            "answer": answer,
            "results": results,
            "citations": citations,
            "tool_usage": response.get("tool_usage") or {"social_search_calls": 1},
        }
        if warning:
            normalized["warning"] = warning
        return normalized

    def _filter_social_results_by_date(
        self,
        results: list[dict[str, Any]],
        *,
        from_date: str | None,
        to_date: str | None,
    ) -> list[dict[str, Any]]:
        if not from_date and not to_date:
            return results

        start = self._parse_date_bound(from_date, end_of_day=False) if from_date else None
        end = self._parse_date_bound(to_date, end_of_day=True) if to_date else None
        filtered: list[dict[str, Any]] = []
        for item in results:
            created_at = self._parse_result_timestamp(item.get("created_at"))
            if created_at is None:
                continue
            if start is not None and created_at < start:
                continue
            if end is not None and created_at > end:
                continue
            filtered.append(item)
        return filtered

    def _parse_date_bound(self, value: str, *, end_of_day: bool) -> datetime | None:
        try:
            parsed = date.fromisoformat(value)
        except ValueError:
            return None
        bound_time = dt_time.max if end_of_day else dt_time.min
        return datetime.combine(parsed, bound_time).replace(tzinfo=timezone.utc)

    def _parse_result_timestamp(self, value: Any) -> datetime | None:
        if not isinstance(value, str) or not value.strip():
            return None
        normalized = value.strip().replace("Z", "+00:00")
        try:
            parsed = datetime.fromisoformat(normalized)
        except ValueError:
            return None
        if parsed.tzinfo is None:
            return parsed.replace(tzinfo=timezone.utc)
        return parsed.astimezone(timezone.utc)

    def _extract_social_gateway_results(self, response: dict[str, Any]) -> list[Any]:
        for key in ("results", "items", "posts", "tweets"):
            value = response.get(key)
            if isinstance(value, list):
                return value

        data = response.get("data")
        if isinstance(data, list):
            return data
        if isinstance(data, dict):
            for key in ("results", "items", "posts", "tweets"):
                value = data.get(key)
                if isinstance(value, list):
                    return value
        return []

    def _extract_social_gateway_citations(
        self,
        response: dict[str, Any],
        results: list[dict[str, Any]],
    ) -> list[dict[str, Any]]:
        if not results:
            return []

        raw = response.get("citations") or response.get("sources") or []
        citations = []
        seen: set[str] = set()
        allowed_urls = {
            item.get("url", "")
            for item in results
            if isinstance(item, dict) and item.get("url")
        }

        if isinstance(raw, list):
            for item in raw:
                citation = self._normalize_citation(item)
                if citation is None:
                    continue
                url = citation.get("url", "")
                if allowed_urls and url and url not in allowed_urls:
                    continue
                if url and url in seen:
                    continue
                if url:
                    seen.add(url)
                citations.append(citation)

        if citations:
            return citations

        for item in results:
            url = item.get("url", "")
            if not url or url in seen:
                continue
            seen.add(url)
            citations.append({"title": item.get("title", ""), "url": url})

        return citations

    def _merge_search_payloads(
        self,
        *,
        primary_result: dict[str, Any],
        secondary_result: dict[str, Any] | None,
        max_results: int,
    ) -> dict[str, Any]:
        sequences: list[list[str]] = []
        variants_by_key: dict[str, list[dict[str, Any]]] = {}
        providers_by_key: dict[str, set[str]] = {}

        for result in [primary_result, secondary_result]:
            if not result:
                continue

            sequence: list[str] = []
            result_provider = result.get("provider", "")
            for item in result.get("results", []) or []:
                if not isinstance(item, dict):
                    continue
                dedupe_key = self._result_dedupe_key(item)
                if not dedupe_key:
                    continue
                sequence.append(dedupe_key)
                variants_by_key.setdefault(dedupe_key, []).append(dict(item))
                providers_by_key.setdefault(dedupe_key, set()).add(
                    item.get("provider") or result_provider
                )
            sequences.append(sequence)

        merged_keys: list[str] = []
        indexes = [0 for _ in sequences]
        seen_keys: set[str] = set()
        while len(merged_keys) < max_results and sequences:
            progressed = False
            for seq_index, sequence in enumerate(sequences):
                while indexes[seq_index] < len(sequence):
                    dedupe_key = sequence[indexes[seq_index]]
                    indexes[seq_index] += 1
                    if dedupe_key in seen_keys:
                        continue
                    seen_keys.add(dedupe_key)
                    merged_keys.append(dedupe_key)
                    progressed = True
                    break
            if not progressed:
                break

        results: list[dict[str, Any]] = []
        matched_results = 0
        for dedupe_key in merged_keys:
            variants = variants_by_key.get(dedupe_key, [])
            if not variants:
                continue
            providers = sorted(item for item in providers_by_key.get(dedupe_key, set()) if item)
            if len(providers) > 1:
                matched_results += 1
            best = max(variants, key=self._result_quality_score)
            merged_item = dict(best)
            merged_item["matched_providers"] = providers
            results.append(merged_item)

        citations = self._dedupe_citations(
            primary_result.get("citations") or [],
            secondary_result.get("citations") or [] if secondary_result else [],
        )
        return {
            "results": results,
            "citations": citations,
            "matched_results": matched_results,
        }

    def _should_rerank_resource_results(
        self,
        *,
        mode: SearchMode,
        intent: ResolvedSearchIntent,
    ) -> bool:
        return mode in {"docs", "github", "pdf"} or intent in {"resource", "tutorial"}

    def _rerank_resource_results(
        self,
        *,
        query: str,
        mode: SearchMode,
        results: list[dict[str, Any]],
        include_domains: list[str] | None,
    ) -> list[dict[str, Any]]:
        if len(results) < 2:
            return results

        query_tokens = self._query_brand_tokens(query)
        ranked = sorted(
            enumerate(results),
            key=lambda pair: (
                self._resource_result_rank(
                    mode=mode,
                    item=pair[1],
                    query_tokens=query_tokens,
                    include_domains=include_domains,
                ),
                -pair[0],
            ),
            reverse=True,
        )
        return [dict(pair[1]) for pair in ranked]

    def _resource_result_rank(
        self,
        *,
        mode: SearchMode,
        item: dict[str, Any],
        query_tokens: list[str],
        include_domains: list[str] | None,
    ) -> tuple[int, int, int, int, int, int, int, int, int, int, int]:
        url = item.get("url", "")
        hostname = self._result_hostname(item)
        registered_domain = self._registered_domain(hostname)
        title_text = (item.get("title") or "").lower()
        include_match = int(
            bool(include_domains)
            and any(self._domain_matches(hostname, domain) for domain in include_domains or [])
        )
        host_brand_match = int(
            any(token in hostname or token in registered_domain for token in query_tokens)
        )
        title_brand_match = int(any(token in title_text for token in query_tokens))
        docs_shape_match = int(
            self._looks_like_resource_result(
                url=url,
                hostname=hostname,
                title_text=title_text,
                mode=mode,
            )
        )
        github_bonus = int(
            mode == "github"
            and hostname in {"github.com", "raw.githubusercontent.com"}
        )
        pdf_bonus = int(mode == "pdf" and self._looks_like_pdf_url(url))
        non_third_party = int(
            not self._is_obvious_third_party_resource(
                hostname=hostname,
                registered_domain=registered_domain,
                mode=mode,
            )
        )
        matched_provider_count = len(item.get("matched_providers") or [])
        content_score, snippet_score, title_score = self._result_quality_score(item)
        return (
            include_match,
            github_bonus,
            pdf_bonus,
            host_brand_match,
            docs_shape_match,
            non_third_party,
            title_brand_match,
            matched_provider_count,
            content_score,
            snippet_score,
            title_score,
        )

    def _align_citations_with_results(
        self,
        *,
        results: list[dict[str, Any]],
        citations: list[dict[str, Any]],
    ) -> list[dict[str, Any]]:
        synthesized = [
            {"title": item.get("title", ""), "url": item.get("url", "")}
            for item in results
            if item.get("url")
        ]
        normalized = self._dedupe_citations(citations, synthesized)
        citations_by_url = {
            item.get("url", ""): item
            for item in normalized
            if item.get("url")
        }

        ordered: list[dict[str, Any]] = []
        seen: set[str] = set()
        for result in results:
            url = result.get("url", "")
            citation = citations_by_url.get(url)
            if citation is None:
                continue
            dedupe_key = self._citation_dedupe_key(citation)
            if dedupe_key in seen:
                continue
            seen.add(dedupe_key)
            ordered.append(citation)

        for citation in normalized:
            dedupe_key = self._citation_dedupe_key(citation)
            if dedupe_key in seen:
                continue
            seen.add(dedupe_key)
            ordered.append(citation)
        return ordered

    def _dedupe_citations(self, *citation_lists: list[dict[str, Any]]) -> list[dict[str, Any]]:
        deduped: list[dict[str, Any]] = []
        seen: set[str] = set()
        for citations in citation_lists:
            for item in citations:
                citation = self._normalize_citation(item)
                if citation is None:
                    continue
                dedupe_key = citation.get("url") or citation.get("title") or json.dumps(
                    citation,
                    ensure_ascii=False,
                    sort_keys=True,
                )
                if dedupe_key in seen:
                    continue
                seen.add(dedupe_key)
                deduped.append(citation)
        return deduped

    def _citation_dedupe_key(self, item: dict[str, Any]) -> str:
        return (
            item.get("url")
            or item.get("title")
            or json.dumps(item, ensure_ascii=False, sort_keys=True)
        )

    def _result_dedupe_key(self, item: dict[str, Any]) -> str:
        url = (item.get("url") or "").strip().lower()
        if url:
            return url
        title = re.sub(r"\s+", " ", (item.get("title") or "").strip().lower())
        snippet = re.sub(r"\s+", " ", (item.get("snippet") or "").strip().lower())
        return f"{title}|{snippet[:160]}".strip("|")

    def _result_quality_score(self, item: dict[str, Any]) -> tuple[int, int, int]:
        content = item.get("content") or ""
        snippet = item.get("snippet") or ""
        title = item.get("title") or ""
        return (len(content), len(snippet), len(title))

    def _result_hostname(self, item: dict[str, Any]) -> str:
        url = (item.get("url") or "").strip()
        if not url:
            return ""
        return self._clean_hostname(urlparse(url).netloc)

    def _clean_hostname(self, hostname: str) -> str:
        cleaned = hostname.lower().strip().strip(".")
        if cleaned.startswith("www."):
            return cleaned[4:]
        return cleaned

    def _registered_domain(self, hostname: str) -> str:
        cleaned = self._clean_hostname(hostname)
        if not cleaned:
            return ""
        parts = cleaned.split(".")
        if len(parts) <= 2:
            return cleaned
        if (
            len(parts) >= 3
            and len(parts[-1]) == 2
            and parts[-2] in {"ac", "co", "com", "edu", "gov", "net", "org"}
        ):
            return ".".join(parts[-3:])
        return ".".join(parts[-2:])

    def _domain_matches(self, hostname: str, domain: str) -> bool:
        cleaned_host = self._clean_hostname(hostname)
        cleaned_domain = self._clean_hostname(domain)
        return bool(cleaned_host) and bool(cleaned_domain) and (
            cleaned_host == cleaned_domain or cleaned_host.endswith(f".{cleaned_domain}")
        )

    def _query_brand_tokens(self, query: str) -> list[str]:
        stopwords = {
            "a",
            "an",
            "and",
            "api",
            "apis",
            "best",
            "changelog",
            "compare",
            "comparison",
            "developer",
            "developers",
            "docs",
            "documentation",
            "for",
            "github",
            "guide",
            "how",
            "manual",
            "pricing",
            "reference",
            "release",
            "releases",
            "sdk",
            "status",
            "the",
            "tutorial",
            "vs",
            "with",
            "价格",
            "发布",
            "对比",
            "接口",
            "教程",
            "文档",
            "更新日志",
        }
        tokens: list[str] = []
        for token in re.findall(r"[a-z0-9][a-z0-9._-]{1,}", query.lower()):
            if token in stopwords or token.isdigit():
                continue
            if len(token) < 3:
                continue
            tokens.append(token)
        return tokens

    def _looks_like_resource_result(
        self,
        *,
        url: str,
        hostname: str,
        title_text: str,
        mode: SearchMode,
    ) -> bool:
        parsed = urlparse(url)
        path = parsed.path.lower()
        hostname_labels = [item for item in hostname.split(".") if item]
        docs_keywords = (
            "/api",
            "/changelog",
            "/docs",
            "/documentation",
            "/guide",
            "/guides",
            "/manual",
            "/pricing",
            "/readme",
            "/reference",
            "/references",
        )
        title_keywords = (
            "api reference",
            "changelog",
            "docs",
            "documentation",
            "guide",
            "manual",
            "pricing",
            "readme",
            "reference",
        )
        hostname_keywords = {
            "api",
            "developer",
            "developers",
            "docs",
            "help",
            "platform",
            "reference",
            "support",
        }
        if mode == "github" and hostname in {"github.com", "raw.githubusercontent.com"}:
            return True
        if mode == "pdf" and self._looks_like_pdf_url(url):
            return True
        return (
            any(part in hostname_keywords for part in hostname_labels)
            or any(keyword in path for keyword in docs_keywords)
            or any(keyword in title_text for keyword in title_keywords)
        )

    def _looks_like_pdf_url(self, url: str) -> bool:
        return urlparse(url).path.lower().endswith(".pdf")

    def _is_obvious_third_party_resource(
        self,
        *,
        hostname: str,
        registered_domain: str,
        mode: SearchMode,
    ) -> bool:
        if mode == "github" and hostname in {"github.com", "raw.githubusercontent.com"}:
            return False
        third_party_domains = {
            "arxiv.org",
            "dev.to",
            "facebook.com",
            "hashnode.dev",
            "hashnode.com",
            "linkedin.com",
            "medium.com",
            "news.ycombinator.com",
            "quora.com",
            "reddit.com",
            "researchgate.net",
            "stackexchange.com",
            "stackoverflow.com",
            "substack.com",
            "towardsdatascience.com",
            "twitter.com",
            "x.com",
            "youtube.com",
            "youtu.be",
        }
        return registered_domain in third_party_domains

    def _describe_provider(
        self,
        provider: ProviderConfig,
        keyring_info: dict[str, object],
    ) -> dict[str, Any]:
        status = self._probe_provider_status(provider, int(keyring_info["count"]))
        return {
            "base_url": provider.base_url,
            "alternate_base_urls": provider.alternate_base_urls,
            "auth_mode": provider.auth_mode,
            "paths": provider.default_paths,
            "search_mode": provider.search_mode,
            "keys_file": str(provider.keys_file or ""),
            "available_keys": keyring_info["count"],
            "sources": keyring_info["sources"],
            "live_status": status["status"],
            "live_error": status["error"],
            "last_checked_at": status["checked_at"],
        }

    def _get_key_or_raise(self, provider: ProviderConfig):
        record = self.keyring.get_next(provider.name)
        if record is None:
            if provider.name == "xai":
                raise MySearchError(
                    "xAI / Social search is not configured; MySearch can still use "
                    "Tavily + Firecrawl for web/docs/extract. Add "
                    "MYSEARCH_XAI_API_KEY for official xAI, or configure a "
                    "compatible /social/search gateway to enable mode='social'."
                )
            if provider.name == "exa":
                raise MySearchError(
                    "Exa search is not configured. Add MYSEARCH_EXA_API_KEY, "
                    "or point MYSEARCH_EXA_BASE_URL to your proxy / compatible gateway."
                )
            raise MySearchError(f"{provider.name} is not configured")
        return record

    def _request_json(
        self,
        *,
        provider: ProviderConfig,
        method: str,
        path: str,
        payload: dict[str, Any],
        key: str,
        base_url: str | None = None,
        timeout_seconds: int | None = None,
    ) -> dict[str, Any]:
        headers: dict[str, str] = {}
        body = dict(payload)

        if provider.auth_mode == "bearer":
            token = key if not provider.auth_scheme else f"{provider.auth_scheme} {key}"
            headers[provider.auth_header] = token
        elif provider.auth_mode == "body":
            body[provider.auth_field] = key
        else:
            raise MySearchError(f"unsupported auth mode for {provider.name}: {provider.auth_mode}")

        url = f"{(base_url or provider.base_url)}{path}"
        headers.setdefault("Content-Type", "application/json")
        headers.setdefault("User-Agent", "MySearch/0.2")
        request_body = json.dumps(body).encode("utf-8")
        request = Request(
            url,
            data=request_body,
            headers=headers,
            method=method.upper(),
        )

        try:
            with urlopen(request, timeout=timeout_seconds or self.config.timeout_seconds) as response:
                status_code = response.status
                response_text = response.read().decode("utf-8", errors="replace")
        except HTTPError as exc:
            status_code = exc.code
            response_text = exc.read().decode("utf-8", errors="replace")
        except (URLError, OSError) as exc:
            raise MySearchError(str(exc)) from exc

        try:
            data = json.loads(response_text)
        except ValueError as exc:
            if status_code >= 400:
                raise MySearchError(f"HTTP {status_code}: {response_text[:300]}") from exc
            raise MySearchError(f"non-json response from {url}: {response_text[:300]}") from exc

        if status_code >= 400:
            detail = data
            if isinstance(data, dict):
                detail = (
                    data.get("detail")
                    or data.get("error")
                    or data.get("message")
                    or data
                )
            raise MySearchHTTPError(
                provider=provider.name,
                status_code=status_code,
                detail=detail,
                url=url,
            )
        return data

    def _probe_provider_status(
        self,
        provider: ProviderConfig,
        key_count: int,
    ) -> dict[str, str]:
        if key_count <= 0:
            return {
                "status": "not_configured",
                "error": "",
                "checked_at": "",
            }

        record = self.keyring.first(provider.name)
        if record is None:
            return {
                "status": "not_configured",
                "error": "",
                "checked_at": "",
            }

        cache_key = f"{provider.name}:{record.label}"
        with self._cache_lock:
            now = time.monotonic()
            cached = self._provider_probe_cache.get(cache_key)
            if cached and cached.get("expires_at", 0.0) > now:
                return copy.deepcopy(cached["value"])

        checked_at = datetime.now(timezone.utc).isoformat()
        try:
            self._probe_provider_request(provider, record.key)
            result = {
                "status": "ok",
                "error": "",
                "checked_at": checked_at,
            }
        except MySearchHTTPError as exc:
            result = {
                "status": "auth_error" if exc.is_auth_error else "http_error",
                "error": str(exc),
                "checked_at": checked_at,
            }
        except MySearchError as exc:
            result = {
                "status": "network_error",
                "error": str(exc),
                "checked_at": checked_at,
            }

        with self._cache_lock:
            self._provider_probe_cache[cache_key] = {
                "expires_at": time.monotonic() + self._provider_probe_ttl_seconds,
                "value": copy.deepcopy(result),
            }
        return result

    def _probe_provider_request(self, provider: ProviderConfig, key: str) -> None:
        timeout_seconds = min(self.config.timeout_seconds, 10)
        if provider.name == "tavily":
            self._request_json(
                provider=provider,
                method="POST",
                path=provider.path("search"),
                payload={
                    "query": "openai",
                    "max_results": 1,
                    "search_depth": "basic",
                    "topic": "general",
                    "include_answer": False,
                    "include_raw_content": False,
                },
                key=key,
                timeout_seconds=timeout_seconds,
            )
            return
        if provider.name == "firecrawl":
            self._request_json(
                provider=provider,
                method="POST",
                path=provider.path("search"),
                payload={
                    "query": "openai",
                    "limit": 1,
                },
                key=key,
                timeout_seconds=timeout_seconds,
            )
            return
        if provider.name == "exa":
            self._request_json(
                provider=provider,
                method="POST",
                path=provider.path("search"),
                payload={
                    "query": "openai",
                    "numResults": 1,
                },
                key=key,
                timeout_seconds=timeout_seconds,
            )
            return
        if provider.name == "xai":
            if provider.search_mode == "compatible":
                self._request_json(
                    provider=provider,
                    method="POST",
                    path=provider.path("social_search"),
                    payload={
                        "query": "openai",
                        "source": "x",
                        "max_results": 1,
                    },
                    key=key,
                    base_url=provider.base_url_for("social_search"),
                    timeout_seconds=timeout_seconds,
                )
                return
            self._request_json(
                provider=provider,
                method="POST",
                path=provider.path("responses"),
                payload=self._build_xai_responses_payload(
                    query="openai",
                    sources=["x"],
                    max_results=1,
                    include_domains=None,
                    exclude_domains=None,
                    allowed_x_handles=None,
                    excluded_x_handles=None,
                    from_date=None,
                    to_date=None,
                    include_x_images=False,
                    include_x_videos=False,
                ),
                key=key,
                timeout_seconds=timeout_seconds,
            )
            return

    def _summarize_route_error(self, error_text: str) -> str:
        compact = " ".join(error_text.split())
        if len(compact) <= 220:
            return compact
        return f"{compact[:217]}..."

    def _provider_can_serve(self, provider: ProviderConfig) -> bool:
        if not self.keyring.has_provider(provider.name):
            return False
        status = self._probe_provider_status(provider, 1)
        return status["status"] != "auth_error"

    def _extract_xai_output_text(self, payload: dict[str, Any]) -> str:
        if isinstance(payload.get("output_text"), str):
            return payload["output_text"]

        parts: list[str] = []
        for item in payload.get("output", []) or []:
            content = item.get("content")
            if isinstance(content, str):
                parts.append(content)
                continue

            if not isinstance(content, list):
                continue

            for part in content:
                if not isinstance(part, dict):
                    continue

                if isinstance(part.get("text"), str):
                    parts.append(part["text"])
                    continue

                text_obj = part.get("text")
                if isinstance(text_obj, dict) and isinstance(text_obj.get("value"), str):
                    parts.append(text_obj["value"])

        return "\n".join([item for item in parts if item]).strip()

    def _extract_xai_citations(self, payload: dict[str, Any]) -> list[dict[str, Any]]:
        raw_citations = payload.get("citations") or []
        normalized: list[dict[str, Any]] = []
        seen: set[str] = set()

        if isinstance(raw_citations, list):
            for item in raw_citations:
                citation = self._normalize_citation(item)
                if citation is None:
                    continue
                url = citation.get("url", "")
                if url and url in seen:
                    continue
                if url:
                    seen.add(url)
                normalized.append(citation)

        if normalized:
            return normalized

        for output_item in payload.get("output", []) or []:
            if not isinstance(output_item, dict):
                continue

            content_items = output_item.get("content") or []
            if not isinstance(content_items, list):
                continue

            for content_item in content_items:
                if not isinstance(content_item, dict):
                    continue

                annotations = content_item.get("annotations") or []
                if not isinstance(annotations, list):
                    continue

                for annotation in annotations:
                    citation = self._normalize_citation(annotation)
                    if citation is None:
                        continue
                    url = citation.get("url", "")
                    if url and url in seen:
                        continue
                    if url:
                        seen.add(url)
                    normalized.append(citation)

        return normalized

    def _normalize_citation(self, item: Any) -> dict[str, Any] | None:
        if not isinstance(item, dict):
            return None

        url = (
            item.get("url")
            or item.get("target_url")
            or item.get("link")
            or item.get("source_url")
            or ""
        )
        title = (
            item.get("title")
            or item.get("source_title")
            or item.get("display_text")
            or item.get("text")
            or ""
        )

        if not url and not title:
            return None

        normalized = dict(item)
        normalized["url"] = url
        normalized["title"] = title
        return normalized

    def _firecrawl_categories(
        self,
        mode: SearchMode,
        intent: ResolvedSearchIntent | None = None,
    ) -> list[str]:
        if mode == "github":
            return ["github"]
        if mode == "pdf":
            return ["pdf"]
        if mode in {"docs", "research"} or intent in {"resource", "tutorial"}:
            return ["research"]
        return []

    def _looks_like_news_query(self, query_lower: str) -> bool:
        keywords = [
            "latest",
            "breaking",
            "news",
            "today",
            "this week",
            "刚刚",
            "最新",
            "新闻",
            "动态",
        ]
        return any(keyword in query_lower for keyword in keywords)

    def _looks_like_status_query(self, query_lower: str) -> bool:
        keywords = [
            "status",
            "incident",
            "outage",
            "release",
            "roadmap",
            "version",
            "版本",
            "发布",
            "进展",
            "现状",
        ]
        return any(keyword in query_lower for keyword in keywords)

    def _looks_like_comparison_query(self, query_lower: str) -> bool:
        keywords = [
            " vs ",
            "versus",
            "compare",
            "comparison",
            "pros and cons",
            "pros cons",
            "对比",
            "比较",
            "区别",
            "哪个好",
        ]
        return any(keyword in query_lower for keyword in keywords)

    def _looks_like_tutorial_query(self, query_lower: str) -> bool:
        keywords = [
            "how to",
            "guide",
            "tutorial",
            "walkthrough",
            "教程",
            "怎么",
            "如何",
            "入门",
        ]
        return any(keyword in query_lower for keyword in keywords)

    def _looks_like_docs_query(self, query_lower: str) -> bool:
        keywords = [
            "docs",
            "documentation",
            "api reference",
            "changelog",
            "pricing",
            "readme",
            "github",
            "manual",
            "文档",
            "接口",
            "价格",
            "更新日志",
        ]
        return any(keyword in query_lower for keyword in keywords)

    def _looks_like_exploratory_query(self, query_lower: str) -> bool:
        keywords = [
            "why",
            "impact",
            "analysis",
            "trend",
            "ecosystem",
            "研究",
            "原因",
            "影响",
            "趋势",
            "生态",
        ]
        return any(keyword in query_lower for keyword in keywords)

    def _build_excerpt(self, content: str, limit: int = 600) -> str:
        compact = re.sub(r"\s+", " ", content).strip()
        if len(compact) <= limit:
            return compact
        return compact[:limit].rstrip() + "..."


path: runtime/mysearch/config.py
content: """MySearch 通用配置。"""

from __future__ import annotations

import os
from dataclasses import dataclass, field
from pathlib import Path
from typing import Literal

try:
    import tomllib  # type: ignore[attr-defined]
except ModuleNotFoundError:  # pragma: no cover - py310 fallback
    tomllib = None  # type: ignore[assignment]


MODULE_DIR = Path(__file__).resolve().parent
ROOT_DIR = MODULE_DIR.parent
AuthMode = Literal["bearer", "body"]
XAISearchMode = Literal["official", "compatible"]
MCPTransport = Literal["stdio", "sse", "streamable-http"]


def _load_env_file(env_path: Path) -> None:
    if not env_path.exists():
        return

    for raw_line in env_path.read_text(encoding="utf-8").splitlines():
        line = raw_line.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue

        key, value = line.split("=", 1)
        key = key.strip()
        value = value.strip()
        if value[:1] == value[-1:] and value[:1] in {"'", '"'}:
            value = value[1:-1]
        os.environ.setdefault(key, value)


def _load_mapping_env(raw_env: dict[str, object]) -> None:
    for key, value in raw_env.items():
        if not isinstance(value, str):
            continue
        cleaned = value.strip()
        if not cleaned:
            continue
        os.environ.setdefault(key, cleaned)


def _parse_codex_mysearch_env(config_text: str) -> dict[str, str]:
    if tomllib is not None:
        try:
            data = tomllib.loads(config_text)
            env = ((data.get("mcp_servers") or {}).get("mysearch") or {}).get("env") or {}
            if isinstance(env, dict):
                return {
                    key: value.strip()
                    for key, value in env.items()
                    if isinstance(value, str) and value.strip()
                }
        except Exception:
            pass

    env: dict[str, str] = {}
    in_section = False
    for raw_line in config_text.splitlines():
        line = raw_line.split("#", 1)[0].strip()
        if not line:
            continue
        if line.startswith("[") and line.endswith("]"):
            in_section = line == "[mcp_servers.mysearch.env]"
            continue
        if not in_section or "=" not in line:
            continue
        key, value = line.split("=", 1)
        key = key.strip()
        value = value.strip()
        if value[:1] == value[-1:] and value[:1] in {"'", '"'}:
            value = value[1:-1]
        if key and value:
            env[key] = value
    return env


def _load_codex_mcp_env() -> None:
    config_path = Path(os.getenv("CODEX_HOME", "~/.codex")).expanduser() / "config.toml"
    if not config_path.exists():
        return

    try:
        env = _parse_codex_mysearch_env(config_path.read_text(encoding="utf-8"))
    except Exception:
        return

    _load_mapping_env(env)


def _load_dotenv() -> None:
    # .env 只作为本地单仓调试兜底，不覆盖宿主已注入的配置。
    for env_path in (MODULE_DIR / ".env", ROOT_DIR / ".env"):
        _load_env_file(env_path)


def _bootstrap_runtime_env() -> None:
    _load_codex_mcp_env()
    _load_dotenv()


def _get_str(*names: str, default: str = "") -> str:
    for name in names:
        value = os.getenv(name)
        if value is not None and value.strip():
            return value.strip()
    return default


def _get_int(name: str, default: int) -> int:
    value = os.getenv(name)
    if value is None or not value.strip():
        return default
    return int(value.strip())


def _get_bool(name: str, default: bool = False) -> bool:
    value = os.getenv(name)
    if value is None:
        return default
    return value.strip().lower() in {"1", "true", "yes", "on"}


def _get_list(*names: str) -> list[str]:
    for name in names:
        value = os.getenv(name)
        if value is None or not value.strip():
            continue
        return [item.strip() for item in value.split(",") if item.strip()]
    return []


def _normalize_base_url(url: str) -> str:
    return url.rstrip("/")


def _normalize_path(path: str) -> str:
    if not path:
        return ""
    if not path.startswith("/"):
        return f"/{path}"
    return path


def _resolve_path(*names: str, default_name: str | None = None) -> Path | None:
    raw = _get_str(*names)
    if raw:
        candidate = Path(raw).expanduser()
        if not candidate.is_absolute():
            candidate = ROOT_DIR / candidate
        return candidate

    if default_name:
        candidate = ROOT_DIR / default_name
        if candidate.exists():
            return candidate
    return None


def _provider_base_url(
    *,
    explicit_names: tuple[str, ...],
    proxy_base_url: str,
    default: str,
) -> str:
    explicit = _get_str(*explicit_names)
    if explicit:
        return _normalize_base_url(explicit)
    if proxy_base_url:
        return _normalize_base_url(proxy_base_url)
    return _normalize_base_url(default)


def _provider_path(
    *,
    explicit_name: str,
    proxy_base_url: str,
    proxy_default: str,
    default: str,
) -> str:
    explicit = _get_str(explicit_name)
    if explicit:
        return _normalize_path(explicit)
    if proxy_base_url:
        return _normalize_path(proxy_default)
    return _normalize_path(default)


_bootstrap_runtime_env()


@dataclass(slots=True)
class ProviderConfig:
    name: str
    base_url: str
    auth_mode: AuthMode
    auth_header: str
    auth_scheme: str
    auth_field: str
    default_paths: dict[str, str]
    alternate_base_urls: dict[str, str] = field(default_factory=dict)
    search_mode: XAISearchMode = "official"
    api_keys: list[str] = field(default_factory=list)
    keys_file: Path | None = None

    def path(self, key: str) -> str:
        return self.default_paths.get(key, "")

    def base_url_for(self, key: str) -> str:
        return self.alternate_base_urls.get(key) or self.base_url


@dataclass(slots=True)
class MySearchConfig:
    server_name: str
    timeout_seconds: int
    xai_model: str
    max_parallel_workers: int
    search_cache_ttl_seconds: int
    extract_cache_ttl_seconds: int
    mcp_host: str
    mcp_port: int
    mcp_mount_path: str
    mcp_sse_path: str
    mcp_streamable_http_path: str
    mcp_stateless_http: bool
    tavily: ProviderConfig
    firecrawl: ProviderConfig
    exa: ProviderConfig
    xai: ProviderConfig

    @classmethod
    def from_env(cls) -> "MySearchConfig":
        proxy_base_url = _get_str("MYSEARCH_PROXY_BASE_URL")
        proxy_api_key = _get_str("MYSEARCH_PROXY_API_KEY")
        return cls(
            server_name=_get_str("MYSEARCH_NAME", "MYSEARCH_SERVER_NAME", default="MySearch"),
            timeout_seconds=_get_int("MYSEARCH_TIMEOUT_SECONDS", 45),
            xai_model=_get_str(
                "MYSEARCH_XAI_MODEL",
                default="grok-4.20-beta-latest-non-reasoning",
            ),
            max_parallel_workers=max(1, _get_int("MYSEARCH_MAX_PARALLEL_WORKERS", 4)),
            search_cache_ttl_seconds=max(0, _get_int("MYSEARCH_SEARCH_CACHE_TTL_SECONDS", 30)),
            extract_cache_ttl_seconds=max(0, _get_int("MYSEARCH_EXTRACT_CACHE_TTL_SECONDS", 300)),
            mcp_host=_get_str("MYSEARCH_MCP_HOST", default="127.0.0.1"),
            mcp_port=_get_int("MYSEARCH_MCP_PORT", 8000),
            mcp_mount_path=_normalize_path(_get_str("MYSEARCH_MCP_MOUNT_PATH", default="/")),
            mcp_sse_path=_normalize_path(_get_str("MYSEARCH_MCP_SSE_PATH", default="/sse")),
            mcp_streamable_http_path=_normalize_path(
                _get_str("MYSEARCH_MCP_STREAMABLE_HTTP_PATH", default="/mcp")
            ),
            mcp_stateless_http=_get_bool("MYSEARCH_MCP_STATELESS_HTTP", False),
            tavily=ProviderConfig(
                name="tavily",
                base_url=_provider_base_url(
                    explicit_names=("MYSEARCH_TAVILY_BASE_URL",),
                    proxy_base_url=proxy_base_url,
                    default="https://api.tavily.com",
                ),
                auth_mode=_get_str(
                    "MYSEARCH_TAVILY_AUTH_MODE",
                    default="bearer" if proxy_base_url else "body",
                ),  # type: ignore[arg-type]
                auth_header=_get_str("MYSEARCH_TAVILY_AUTH_HEADER", default="Authorization"),
                auth_scheme=_get_str("MYSEARCH_TAVILY_AUTH_SCHEME", default="Bearer"),
                auth_field=_get_str("MYSEARCH_TAVILY_AUTH_FIELD", default="api_key"),
                default_paths={
                    "search": _provider_path(
                        explicit_name="MYSEARCH_TAVILY_SEARCH_PATH",
                        proxy_base_url=proxy_base_url,
                        proxy_default="/api/search",
                        default="/search",
                    ),
                    "extract": _provider_path(
                        explicit_name="MYSEARCH_TAVILY_EXTRACT_PATH",
                        proxy_base_url=proxy_base_url,
                        proxy_default="/api/extract",
                        default="/extract",
                    ),
                },
                api_keys=[
                    *_get_list("MYSEARCH_TAVILY_API_KEYS"),
                    *(
                        [_get_str("MYSEARCH_TAVILY_API_KEY")]
                        if _get_str("MYSEARCH_TAVILY_API_KEY")
                        else ([proxy_api_key] if proxy_api_key else [])
                    ),
                ],
                keys_file=_resolve_path(
                    "MYSEARCH_TAVILY_KEYS_FILE",
                    "MYSEARCH_TAVILY_ACCOUNTS_FILE",
                    default_name="accounts.txt",
                ),
            ),
            firecrawl=ProviderConfig(
                name="firecrawl",
                base_url=_provider_base_url(
                    explicit_names=("MYSEARCH_FIRECRAWL_BASE_URL",),
                    proxy_base_url=proxy_base_url,
                    default="https://api.firecrawl.dev",
                ),
                auth_mode=_get_str("MYSEARCH_FIRECRAWL_AUTH_MODE", default="bearer"),  # type: ignore[arg-type]
                auth_header=_get_str("MYSEARCH_FIRECRAWL_AUTH_HEADER", default="Authorization"),
                auth_scheme=_get_str("MYSEARCH_FIRECRAWL_AUTH_SCHEME", default="Bearer"),
                auth_field=_get_str("MYSEARCH_FIRECRAWL_AUTH_FIELD", default="api_key"),
                default_paths={
                    "search": _provider_path(
                        explicit_name="MYSEARCH_FIRECRAWL_SEARCH_PATH",
                        proxy_base_url=proxy_base_url,
                        proxy_default="/firecrawl/v2/search",
                        default="/v2/search",
                    ),
                    "scrape": _provider_path(
                        explicit_name="MYSEARCH_FIRECRAWL_SCRAPE_PATH",
                        proxy_base_url=proxy_base_url,
                        proxy_default="/firecrawl/v2/scrape",
                        default="/v2/scrape",
                    ),
                },
                api_keys=[
                    *_get_list("MYSEARCH_FIRECRAWL_API_KEYS"),
                    *(
                        [_get_str("MYSEARCH_FIRECRAWL_API_KEY")]
                        if _get_str("MYSEARCH_FIRECRAWL_API_KEY")
                        else ([proxy_api_key] if proxy_api_key else [])
                    ),
                ],
                keys_file=_resolve_path(
                    "MYSEARCH_FIRECRAWL_KEYS_FILE",
                    "MYSEARCH_FIRECRAWL_ACCOUNTS_FILE",
                    default_name="firecrawl_accounts.txt",
                ),
            ),
            exa=ProviderConfig(
                name="exa",
                base_url=_provider_base_url(
                    explicit_names=("MYSEARCH_EXA_BASE_URL",),
                    proxy_base_url=proxy_base_url,
                    default="https://api.exa.ai",
                ),
                auth_mode=_get_str("MYSEARCH_EXA_AUTH_MODE", default="bearer"),  # type: ignore[arg-type]
                auth_header=_get_str(
                    "MYSEARCH_EXA_AUTH_HEADER",
                    default="Authorization" if proxy_base_url else "x-api-key",
                ),
                auth_scheme=_get_str(
                    "MYSEARCH_EXA_AUTH_SCHEME",
                    default="Bearer" if proxy_base_url else "",
                ),
                auth_field=_get_str("MYSEARCH_EXA_AUTH_FIELD", default="api_key"),
                default_paths={
                    "search": _provider_path(
                        explicit_name="MYSEARCH_EXA_SEARCH_PATH",
                        proxy_base_url=proxy_base_url,
                        proxy_default="/exa/search",
                        default="/search",
                    ),
                },
                api_keys=[
                    *_get_list("MYSEARCH_EXA_API_KEYS"),
                    *(
                        [_get_str("MYSEARCH_EXA_API_KEY")]
                        if _get_str("MYSEARCH_EXA_API_KEY")
                        else ([proxy_api_key] if proxy_api_key else [])
                    ),
                ],
                keys_file=_resolve_path(
                    "MYSEARCH_EXA_KEYS_FILE",
                    "MYSEARCH_EXA_ACCOUNTS_FILE",
                    default_name="exa_accounts.txt",
                ),
            ),
            xai=ProviderConfig(
                name="xai",
                base_url=_normalize_base_url(
                    _get_str("MYSEARCH_XAI_BASE_URL", default="https://api.x.ai/v1")
                ),
                auth_mode=_get_str("MYSEARCH_XAI_AUTH_MODE", default="bearer"),  # type: ignore[arg-type]
                auth_header=_get_str("MYSEARCH_XAI_AUTH_HEADER", default="Authorization"),
                auth_scheme=_get_str("MYSEARCH_XAI_AUTH_SCHEME", default="Bearer"),
                auth_field=_get_str("MYSEARCH_XAI_AUTH_FIELD", default="api_key"),
                default_paths={
                    "responses": _normalize_path(
                        _get_str("MYSEARCH_XAI_RESPONSES_PATH", default="/responses")
                    ),
                    "social_search": _provider_path(
                        explicit_name="MYSEARCH_XAI_SOCIAL_SEARCH_PATH",
                        proxy_base_url=proxy_base_url,
                        proxy_default="/social/search",
                        default="/social/search",
                    ),
                },
                alternate_base_urls={
                    "social_search": _normalize_base_url(
                        _get_str("MYSEARCH_XAI_SOCIAL_BASE_URL") or proxy_base_url
                    )
                },
                search_mode=_get_str(
                    "MYSEARCH_XAI_SEARCH_MODE",
                    default="compatible" if proxy_base_url else "official",
                ),  # type: ignore[arg-type]
                api_keys=[
                    *_get_list("MYSEARCH_XAI_API_KEYS"),
                    *(
                        [_get_str("MYSEARCH_XAI_API_KEY")]
                        if _get_str("MYSEARCH_XAI_API_KEY")
                        else ([proxy_api_key] if proxy_api_key else [])
                    ),
                ],
                keys_file=_resolve_path("MYSEARCH_XAI_KEYS_FILE"),
            ),
        )


path: runtime/mysearch/keyring.py
content: """MySearch 通用 key ring。"""

from __future__ import annotations

from dataclasses import dataclass
from threading import Lock

from mysearch.config import MySearchConfig, ProviderConfig


@dataclass(frozen=True, slots=True)
class KeyRecord:
    provider: str
    key: str
    source: str
    label: str


class MySearchKeyRing:
    def __init__(self, config: MySearchConfig) -> None:
        self.config = config
        self._lock = Lock()
        self._keys: dict[str, list[KeyRecord]] = {
            "tavily": [],
            "firecrawl": [],
            "exa": [],
            "xai": [],
        }
        self._indexes = {
            "tavily": 0,
            "firecrawl": 0,
            "exa": 0,
            "xai": 0,
        }
        self.reload()

    def reload(self) -> None:
        with self._lock:
            self._keys["tavily"] = self._load_provider(self.config.tavily)
            self._keys["firecrawl"] = self._load_provider(self.config.firecrawl)
            self._keys["exa"] = self._load_provider(self.config.exa)
            self._keys["xai"] = self._load_provider(self.config.xai)
            for provider, keys in self._keys.items():
                if self._indexes[provider] >= len(keys):
                    self._indexes[provider] = 0

    def get_next(self, provider: str) -> KeyRecord | None:
        with self._lock:
            keys = self._keys[provider]
            if not keys:
                return None
            index = self._indexes[provider]
            self._indexes[provider] = (index + 1) % len(keys)
            return keys[index]

    def has_provider(self, provider: str) -> bool:
        with self._lock:
            return bool(self._keys.get(provider))

    def first(self, provider: str) -> KeyRecord | None:
        with self._lock:
            keys = self._keys.get(provider) or []
            if not keys:
                return None
            return keys[0]

    def describe(self) -> dict[str, dict[str, object]]:
        with self._lock:
            result: dict[str, dict[str, object]] = {}
            for provider, keys in self._keys.items():
                result[provider] = {
                    "count": len(keys),
                    "sources": sorted({key.source for key in keys}),
                    "labels": [key.label for key in keys],
                }
            return result

    def _load_provider(self, provider: ProviderConfig) -> list[KeyRecord]:
        loaded: list[KeyRecord] = []

        for index, key in enumerate(provider.api_keys, start=1):
            cleaned = key.strip()
            if not cleaned:
                continue
            loaded.append(
                KeyRecord(
                    provider=provider.name,
                    key=cleaned,
                    source="env",
                    label=f"{provider.name}:env:{index}",
                )
            )

        if provider.keys_file and provider.keys_file.exists():
            loaded.extend(self._load_from_file(provider))

        deduped: list[KeyRecord] = []
        seen: set[str] = set()
        for record in loaded:
            if record.key in seen:
                continue
            seen.add(record.key)
            deduped.append(record)
        return deduped

    def _load_from_file(self, provider: ProviderConfig) -> list[KeyRecord]:
        records: list[KeyRecord] = []
        assert provider.keys_file is not None

        for line_no, raw_line in enumerate(
            provider.keys_file.read_text(encoding="utf-8").splitlines(),
            start=1,
        ):
            line = raw_line.strip()
            if not line or line.startswith("#"):
                continue

            parts = [item.strip() for item in line.split(",") if item.strip()]
            key = parts[-1] if parts else line
            label = parts[0] if len(parts) >= 2 else f"{provider.keys_file.name}:{line_no}"
            if not key:
                continue

            records.append(
                KeyRecord(
                    provider=provider.name,
                    key=key,
                    source="file",
                    label=label,
                )
            )

        return records


path: scripts/install_openclaw_skill.sh
content: #!/usr/bin/env bash
set -euo pipefail

SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
TARGET_DIR="$SOURCE_DIR"
COPY_ENV=""

usage() {
  cat <<'EOF'
Usage: install_openclaw_skill.sh [options]

Prepare the bundled MySearch OpenClaw skill for local use.
This installer does not download remote runtime code and does not modify
other installed skills.

Options:
  --install-to DIR   Copy the skill bundle into DIR before finishing setup
  --copy-env FILE    Copy FILE to target .env with 0600 permissions
                     Optional. Prefer OpenClaw skill env injection instead.
  -h, --help         Show this help

Examples:
  bash scripts/install_openclaw_skill.sh
  bash scripts/install_openclaw_skill.sh --install-to ~/.openclaw/skills/mysearch
  bash scripts/install_openclaw_skill.sh --copy-env ~/.config/mysearch.env
EOF
}

while [[ $# -gt 0 ]]; do
  case "$1" in
    --install-to)
      TARGET_DIR="${2:?missing dir}"
      shift 2
      ;;
    --copy-env)
      COPY_ENV="${2:?missing env path}"
      shift 2
      ;;
    -h|--help)
      usage
      exit 0
      ;;
    *)
      echo "Unknown option: $1" >&2
      usage >&2
      exit 2
      ;;
  esac
done

copy_skill_bundle() {
  if [[ "$TARGET_DIR" == "$SOURCE_DIR" ]]; then
    return
  fi

  mkdir -p "$TARGET_DIR"
  tar -C "$SOURCE_DIR" \
    --exclude='.env' \
    --exclude='__pycache__' \
    -cf - . | tar -C "$TARGET_DIR" -xf -
}

prepare_env_file() {
  if [[ "$SOURCE_DIR/.env.example" != "$TARGET_DIR/.env.example" ]]; then
    install -m 0644 "$SOURCE_DIR/.env.example" "$TARGET_DIR/.env.example"
  fi

  if [[ -n "$COPY_ENV" ]]; then
    install -m 0600 "$COPY_ENV" "$TARGET_DIR/.env"
  fi
}

copy_skill_bundle
prepare_env_file

cat <<EOF
MySearch OpenClaw skill is ready at: $TARGET_DIR

What changed:
1. Runtime is bundled inside the skill package
2. No remote downloads were performed
3. No other installed skills were modified

Next steps:
1. Prefer injecting env via OpenClaw skill config instead of copying secrets into the skill folder
2. Minimal trusted setup: MYSEARCH_PROXY_BASE_URL + MYSEARCH_PROXY_API_KEY
3. If you do not have a proxy yet, fall back to MYSEARCH_TAVILY_API_KEY + MYSEARCH_FIRECRAWL_API_KEY
4. Only use --copy-env or $TARGET_DIR/.env for local debugging
5. Run: python3 $TARGET_DIR/scripts/mysearch_openclaw.py health
EOF


path: scripts/mysearch_openclaw.py
content: #!/usr/bin/env python3
"""OpenClaw wrapper for MySearch."""

from __future__ import annotations

import argparse
import json
import os
import sys
from pathlib import Path
from typing import Any


BASE_DIR = Path(__file__).resolve().parents[1]


def _load_env_file(path: Path) -> None:
    if not path.exists():
        return

    for raw_line in path.read_text(encoding="utf-8").splitlines():
        line = raw_line.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        key, value = line.split("=", 1)
        key = key.strip()
        value = value.strip()
        if value[:1] == value[-1:] and value[:1] in {"'", '"'}:
            value = value[1:-1]
        os.environ.setdefault(key, value)


def _load_mapping_env(raw_env: dict[str, object]) -> None:
    for key, value in raw_env.items():
        if not isinstance(value, str):
            continue
        cleaned = value.strip()
        if not cleaned:
            continue
        os.environ.setdefault(key, cleaned)


def _load_openclaw_skill_env(base_dir: Path) -> None:
    candidates: list[Path] = []
    explicit = os.getenv("OPENCLAW_CONFIG_PATH", "").strip()
    if explicit:
        candidates.append(Path(explicit).expanduser())
    candidates.append(base_dir.parents[1] / "openclaw.json")

    seen: set[Path] = set()
    for path in candidates:
        if path in seen or not path.exists():
            continue
        seen.add(path)
        try:
            config = json.loads(path.read_text(encoding="utf-8"))
        except Exception:
            continue
        env = (((config.get("skills") or {}).get("entries") or {}).get("mysearch") or {}).get("env") or {}
        if isinstance(env, dict):
            _load_mapping_env(env)
            return


def _bootstrap_error(message: str) -> int:
    print(message, file=sys.stderr)
    print(
        f"Run: bash {BASE_DIR / 'scripts' / 'install_openclaw_skill.sh'}",
        file=sys.stderr,
    )
    return 2


def _runtime_imports():
    runtime_dir = BASE_DIR / "runtime"
    runtime_pkg = runtime_dir / "mysearch"
    if not runtime_pkg.exists():
        raise RuntimeError("MySearch runtime is not installed inside this skill yet.")

    if str(runtime_dir) not in sys.path:
        sys.path.insert(0, str(runtime_dir))

    try:
        from mysearch.clients import MySearchClient, MySearchError  # type: ignore
    except ModuleNotFoundError as exc:  # pragma: no cover - runtime bootstrap path
        raise RuntimeError(
            "MySearch bundled runtime is incomplete. Reinstall the skill package."
        ) from exc

    return MySearchClient, MySearchError


def _parse_csv(raw: str | None) -> list[str] | None:
    if not raw:
        return None
    values = [item.strip() for item in raw.split(",") if item.strip()]
    return values or None


def _snippet(item: dict[str, Any]) -> str:
    for key in ("content", "snippet", "summary", "text", "excerpt"):
        value = (item.get(key) or "").strip()
        if value:
            return value
    return ""


def _result_lines(items: list[dict[str, Any]], heading: str) -> list[str]:
    lines = [heading, ""]
    if not items:
        lines.append("- 无结果")
        lines.append("")
        return lines

    for index, item in enumerate(items, start=1):
        title = (item.get("title") or item.get("url") or f"结果 {index}").strip()
        url = (item.get("url") or "").strip()
        snippet = _snippet(item)
        meta_bits: list[str] = []

        for key in ("source", "published_date", "date", "handle", "author"):
            value = item.get(key)
            if value:
                meta_bits.append(f"{key}={value}")

        lines.append(f"{index}. {title}")
        if url:
            lines.append(f"   {url}")
        if meta_bits:
            lines.append(f"   {' | '.join(meta_bits)}")
        if snippet:
            compact = " ".join(snippet.split())
            if len(compact) > 400:
                compact = f"{compact[:397]}..."
            lines.append(f"   {compact}")
        lines.append("")
    return lines


def _render_nested_search(title: str, payload: dict[str, Any]) -> list[str]:
    lines = [f"## {title}", ""]
    route = payload.get("route") or {}
    if route:
        lines.append(f"- route: `{route.get('selected', '')}`")
        if route.get("reason"):
            lines.append(f"- reason: {route['reason']}")
        lines.append("")

    answer = (payload.get("answer") or "").strip()
    if answer:
        lines.extend(["### Answer", "", answer, ""])

    lines.extend(_result_lines(payload.get("results") or [], "### Results"))
    return lines


def _render_health(payload: dict[str, Any]) -> str:
    lines = [
        "# MySearch Health",
        "",
        f"- server_name: `{payload.get('server_name', '')}`",
        f"- timeout_seconds: `{payload.get('timeout_seconds', '')}`",
        f"- xai_model: `{payload.get('xai_model', '')}`",
        "",
    ]

    for provider_name, info in (payload.get("providers") or {}).items():
        lines.extend(
            [
                f"## {provider_name}",
                "",
                f"- base_url: `{info.get('base_url', '')}`",
                f"- auth_mode: `{info.get('auth_mode', '')}`",
                f"- search_mode: `{info.get('search_mode', '')}`",
                f"- available_keys: `{info.get('available_keys', 0)}`",
                f"- live_status: `{info.get('live_status', '')}`",
            ]
        )
        if info.get("live_error"):
            lines.append(f"- live_error: {info['live_error']}")
        alternate = info.get("alternate_base_urls") or {}
        for key, value in alternate.items():
            if value:
                lines.append(f"- alternate.{key}: `{value}`")
        lines.append("")

    return "\n".join(lines).strip()


def _render_search(payload: dict[str, Any]) -> str:
    lines = [
        "# MySearch Search",
        "",
        f"- query: {payload.get('query', '')}",
        f"- provider: `{payload.get('provider', '')}`",
        f"- intent: `{payload.get('intent', '')}`",
        f"- strategy: `{payload.get('strategy', '')}`",
    ]
    route = payload.get("route") or {}
    if route:
        lines.append(f"- route: `{route.get('selected', '')}`")
        if route.get("reason"):
            lines.append(f"- reason: {route['reason']}")
    lines.append("")

    if payload.get("provider") == "hybrid" and (payload.get("web") or payload.get("social")):
        if payload.get("web"):
            lines.extend(_render_nested_search("Web", payload["web"]))
        if payload.get("social"):
            lines.extend(_render_nested_search("Social / X", payload["social"]))
    else:
        answer = (payload.get("answer") or "").strip()
        if answer:
            lines.extend(["## Answer", "", answer, ""])
        lines.extend(_result_lines(payload.get("results") or [], "## Results"))

    citations = payload.get("citations") or []
    if citations:
        lines.extend(["## Citations", ""])
        for item in citations:
            title = item.get("title") or item.get("url") or "citation"
            url = item.get("url") or ""
            if url:
                lines.append(f"- {title}: {url}")
            else:
                lines.append(f"- {title}")
        lines.append("")

    return "\n".join(lines).strip()


def _render_extract(payload: dict[str, Any]) -> str:
    lines = [
        "# MySearch Extract",
        "",
        f"- url: {payload.get('url', '')}",
        f"- provider: `{payload.get('provider', '')}`",
    ]
    if payload.get("warning"):
        lines.append(f"- warning: {payload['warning']}")
    if payload.get("fallback"):
        fallback = payload["fallback"]
        lines.append(
            f"- fallback: `{fallback.get('from', '')}` -> `{payload.get('provider', '')}` ({fallback.get('reason', '')})"
        )
    lines.extend(["", "## Content", "", payload.get("content", "").strip() or "(empty)"])
    return "\n".join(lines).strip()


def _render_research(payload: dict[str, Any]) -> str:
    lines = [
        "# MySearch Research",
        "",
        f"- query: {payload.get('query', '')}",
        f"- provider: `{payload.get('provider', '')}`",
        f"- intent: `{payload.get('intent', '')}`",
        f"- strategy: `{payload.get('strategy', '')}`",
        "",
    ]

    if payload.get("web_search"):
        lines.extend(_render_nested_search("Web Search", payload["web_search"]))

    pages = payload.get("pages") or []
    lines.extend(["## Pages", ""])
    if not pages:
        lines.append("- 无抓取页面")
        lines.append("")
    else:
        for index, page in enumerate(pages, start=1):
            url = page.get("url") or f"page-{index}"
            provider = page.get("provider") or ""
            excerpt = (page.get("excerpt") or "").strip()
            lines.append(f"{index}. {url}")
            if provider:
                lines.append(f"   provider={provider}")
            if page.get("error"):
                lines.append(f"   error={page['error']}")
            if page.get("warning"):
                lines.append(f"   warning={page['warning']}")
            if excerpt:
                compact = " ".join(excerpt.split())
                if len(compact) > 400:
                    compact = f"{compact[:397]}..."
                lines.append(f"   {compact}")
            lines.append("")

    if payload.get("social_search"):
        lines.extend(_render_nested_search("Social / X", payload["social_search"]))
    elif payload.get("social_error"):
        lines.extend(["## Social / X", "", f"- {payload['social_error']}", ""])

    evidence = payload.get("evidence") or {}
    if evidence:
        lines.extend(["## Evidence", ""])
        for key, value in evidence.items():
            lines.append(f"- {key}: {value}")
        lines.append("")

    return "\n".join(lines).strip()


def main() -> int:
    _load_openclaw_skill_env(BASE_DIR)
    _load_env_file(BASE_DIR / ".env")
    _load_env_file(BASE_DIR / "runtime" / ".env")

    try:
        MySearchClient, MySearchError = _runtime_imports()
    except RuntimeError as exc:
        return _bootstrap_error(str(exc))

    parser = argparse.ArgumentParser(description="OpenClaw wrapper for MySearch")
    parser.add_argument("--format", choices=("md", "json"), default="md")
    subparsers = parser.add_subparsers(dest="command", required=True)

    subparsers.add_parser("health", help="Show provider health")

    search_parser = subparsers.add_parser("search", help="Run a MySearch search")
    search_parser.add_argument("--query", required=True)
    search_parser.add_argument("--mode", default="auto", choices=("auto", "web", "news", "social", "docs", "research", "github", "pdf"))
    search_parser.add_argument("--intent", default="auto", choices=("auto", "factual", "status", "comparison", "tutorial", "exploratory", "news", "resource"))
    search_parser.add_argument("--strategy", default="auto", choices=("auto", "fast", "balanced", "verify", "deep"))
    search_parser.add_argument("--provider", default="auto", choices=("auto", "tavily", "firecrawl", "exa", "xai"))
    search_parser.add_argument("--sources", default="")
    search_parser.add_argument("-n", "--max-results", type=int, default=5)
    search_parser.add_argument("--include-content", action="store_true")
    search_parser.add_argument("--no-answer", action="store_true")
    search_parser.add_argument("--domains", default="")
    search_parser.add_argument("--exclude-domains", default="")
    search_parser.add_argument("--x-handles", default="")
    search_parser.add_argument("--exclude-x-handles", default="")
    search_parser.add_argument("--from-date", default="")
    search_parser.add_argument("--to-date", default="")

    extract_parser = subparsers.add_parser("extract", help="Extract a single URL")
    extract_parser.add_argument("--url", required=True)
    extract_parser.add_argument("--provider", default="auto", choices=("auto", "firecrawl", "tavily"))

    research_parser = subparsers.add_parser("research", help="Run a MySearch research bundle")
    research_parser.add_argument("--query", required=True)
    research_parser.add_argument("--mode", default="auto", choices=("auto", "web", "news", "social", "docs", "research", "github", "pdf"))
    research_parser.add_argument("--intent", default="auto", choices=("auto", "factual", "status", "comparison", "tutorial", "exploratory", "news", "resource"))
    research_parser.add_argument("--strategy", default="auto", choices=("auto", "fast", "balanced", "verify", "deep"))
    research_parser.add_argument("--web-max-results", type=int, default=5)
    research_parser.add_argument("--social-max-results", type=int, default=5)
    research_parser.add_argument("--scrape-top-n", type=int, default=3)
    research_parser.add_argument("--include-social", action="store_true", default=False)
    research_parser.add_argument("--domains", default="")
    research_parser.add_argument("--exclude-domains", default="")
    research_parser.add_argument("--x-handles", default="")
    research_parser.add_argument("--exclude-x-handles", default="")
    research_parser.add_argument("--from-date", default="")
    research_parser.add_argument("--to-date", default="")

    args = parser.parse_args()
    client = MySearchClient()

    try:
        if args.command == "health":
            payload = client.health()
            renderer = _render_health
        elif args.command == "search":
            payload = client.search(
                query=args.query,
                mode=args.mode,
                intent=args.intent,
                strategy=args.strategy,
                provider=args.provider,
                sources=_parse_csv(args.sources),
                max_results=args.max_results,
                include_content=args.include_content,
                include_answer=not args.no_answer,
                include_domains=_parse_csv(args.domains),
                exclude_domains=_parse_csv(args.exclude_domains),
                allowed_x_handles=_parse_csv(args.x_handles),
                excluded_x_handles=_parse_csv(args.exclude_x_handles),
                from_date=args.from_date or None,
                to_date=args.to_date or None,
            )
            renderer = _render_search
        elif args.command == "extract":
            payload = client.extract_url(
                url=args.url,
                formats=["markdown"],
                provider=args.provider,
            )
            renderer = _render_extract
        else:
            payload = client.research(
                query=args.query,
                web_max_results=args.web_max_results,
                social_max_results=args.social_max_results,
                scrape_top_n=args.scrape_top_n,
                include_social=args.include_social,
                mode=args.mode,
                intent=args.intent,
                strategy=args.strategy,
                include_domains=_parse_csv(args.domains),
                exclude_domains=_parse_csv(args.exclude_domains),
                allowed_x_handles=_parse_csv(args.x_handles),
                excluded_x_handles=_parse_csv(args.exclude_x_handles),
                from_date=args.from_date or None,
                to_date=args.to_date or None,
            )
            renderer = _render_research
    except MySearchError as exc:
        print(str(exc), file=sys.stderr)
        return 1

    if args.format == "json":
        print(json.dumps(payload, ensure_ascii=False, indent=2))
    else:
        print(renderer(payload))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)