Skill: vulcanusalex/free-search-aggregator (version 1.3.0)

--- name: free-search-aggregator description: Quota-aware multi-provider web search for OpenClaw. Supports 12 search providers with automatic failover, task-level deep search (@dual/@deep), real quota checks, and managed result storage under memory/. --- # Free Search Aggregator Reliable, provider-diverse web search for OpenClaw with **high uptime + low operator overhead**. ## Why use this skill - **12 search providers**, 6 requiring no API key at all - Automatic failover: if one provider fails, the next is tried instantly - Quota-aware: tracks daily usage, warns at 80%, skips exhausted providers - Task search mode for multi-angle research queries - Built-in storage lifecycle (cache / index / report), no workspace clutter - **Self-healing**: health-based smart routing automatically promotes reliable providers - **Quality optimization**: relevance scoring, fuzzy dedup, domain diversity, re-ranking - **Auto-discovery**: probes candidate search engines and SearXNG instances for new sources - **Self-diagnostic**: `doctor` and `setup` commands for zero-friction onboarding --- ## Provider Overview | Provider | Key Required | Free Quota | Index Source | Notes | |----------------|-------------|-------------------|-----------------------|---------------------------------| | `brave` | BRAVE_API_KEY | 2000/day | Brave independent | High quality, privacy-friendly | | `exa` | EXA_API_KEY | ~33/day (1k/mo) | Neural + web | Semantic search, unique finds | | `tavily` | TAVILY_API_KEY | 1000/day | Web (AI-optimized) | Designed for AI agents | | `duckduckgo` | None | ~500/day | Bing + own | No key, privacy-focused | | `bing_html` | None | ~300/day | Microsoft Bing RSS | No key, stable XML feed | | `mojeek` | None (or MOJEEK_API_KEY) | 200/day | Mojeek independent | Non-Google/Bing index | | `serper` | SERPER_API_KEY | 2500/day | Google | High quota free tier | | `searchapi` | SEARCHAPI_API_KEY | 100/mo | Google / Bing | Multi-engine | | `google_cse` | GOOGLE_API_KEY + GOOGLE_CX | 100/day | Google | Official Google API | | `baidu` | BAIDU_API_KEY | 200/day | Baidu | Best for Chinese content | | `wikipedia` | None | 1000/day | Wikipedia | Factual/encyclopedic queries | | `searxng` | None | unlimited (self-hosted) | Meta (all engines) | Requires own instance | **Total daily quota (all keys configured): 8400+ requests/day** --- ## Credential model (important) - **No mandatory API key** — DuckDuckGo + Bing RSS + Mojeek + Wikipedia work out of the box. - API-key providers fail gracefully if key is missing (AuthError → skip, no quota consumed, no latency): - `BRAVE_API_KEY` - `EXA_API_KEY` - `TAVILY_API_KEY` - `SERPER_API_KEY` - `SEARCHAPI_API_KEY` - `GOOGLE_API_KEY` + `GOOGLE_CX` - `BAIDU_API_KEY` - `MOJEEK_API_KEY` (optional — without it uses HTML scraping) --- ## Core capabilities ### 1. Search failover Default provider order: ``` brave → exa → tavily → duckduckgo → bing_html → mojeek → serper → searchapi → google_cse → baidu → wikipedia ``` First successful non-empty result returns immediately. ### 2. Task-level multi-query search - Expands one goal into multiple targeted queries - Aggregates + deduplicates results - Prefix presets: - default: `workers=1` - `@dual ...` → `workers=2` - `@deep ...` → `workers=3` + deeper query coverage ### 3. Quota intelligence - Per-provider daily tracking - Real quota retrieval where supported (Tavily, SearchAPI, Brave via probe) - Auto concurrency reduction at 80% quota saturation ### 4. Provider health monitoring - Tracks success rate, latency, and error types per provider over time - Computes health scores (success 50%, latency 30%, freshness 20%) - **Smart ordering**: auto-promotes healthy providers, demotes degraded ones - View dashboard: `python -m free_search health` ### 5. Result quality optimization - Relevance scoring (query-title-snippet token overlap) - Enhanced dedup: URL + title similarity (Jaccard threshold) - Domain diversity: limits same-domain results (default max 3) - Automatic filtering of low-quality results (short titles, missing URLs) ### 6. Source auto-discovery - Probes all configured providers for availability - Scans candidate search engines (Marginalia, Wiby, public SearXNG instances) - Validates response format, latency, and result quality - Generates recommendations for new sources to integrate - Run: `python -m free_search discover` ### 7. Managed persistence - `memory/search-cache/YYYY-MM-DD/*.json` - `memory/search-index/search-index.jsonl` - `memory/search-reports/YYYY-MM-DD/*.md` --- ## Quick commands ```bash # Normal search scripts/search "latest AI agent frameworks 2026" --max-results 5 # Task search (multi-query, parallel) scripts/search task "@dual Compare Claude vs GPT-4 for code generation" --max-results 5 # Deep research mode scripts/search task "@deep autonomous vehicle safety 2026" --max-results 8 --max-queries 10 # Quota status scripts/status # Real quota from provider APIs scripts/remaining --real # Cleanup cache python3 -m free_search gc --cache-days 14 # Provider health dashboard python3 -m free_search health # Discover new search sources python3 -m free_search discover # System diagnostics python3 -m free_search doctor # Setup status & recommendations python3 -m free_search setup ``` --- ## Provider setup guides ### Bing RSS (`bing_html`) — No key needed Uses Bing's built-in RSS endpoint (`format=rss`) — bypasses bot detection. Works out of the box. ### Mojeek — No key needed (API key optional) Out-of-the-box HTML scraping. For higher quotas/stability: 1. Register at https://www.mojeek.com/services/search/api/ 2. Set `MOJEEK_API_KEY` → automatically switches to JSON API mode ### Wikipedia — No key needed Multilingual support — change `lang` in `providers.yaml`: ```yaml wikipedia: lang: it # en | zh | it | de | fr | ja ... ``` ### Exa.ai — API key required 1. Register at https://exa.ai/ 2. Set `EXA_API_KEY` 3. Free tier: 1000 searches/month (~33/day) ### Google Custom Search — API key + CX required 1. Get API key: https://developers.google.com/custom-search/v1/introduction 2. Create search engine: https://programmablesearchengine.google.com/ 3. Set `GOOGLE_API_KEY` and `GOOGLE_CX` 4. Free tier: 100 queries/day ### Baidu Qianfan — API key required 1. Register at https://cloud.baidu.com/ 2. Set `BAIDU_API_KEY` 3. Best for Chinese-language content ### SearXNG — Self-hosted instance required Public instances rate-limit server-to-server requests. Use your own: ```bash docker run -d -p 8080:8080 searxng/searxng ``` Then in `providers.yaml`: ```yaml searxng: endpoint: http://localhost:8080 enabled: true ``` --- ## Post-install self-check ```bash # 1) Confirm provider load scripts/status --compact # 2) Smoke test (uses duckduckgo/bing/mojeek out of the box) scripts/search "openclaw" --max-results 3 --compact # 3) Verify storage paths ls -la /home/openclaw/.openclaw/workspace/memory/search-cache/ | tail -n 5 # 4) Check real quota (optional) scripts/remaining --real --compact ``` --- ## Output contract (stable) - **Search**: `query`, `provider`, `results[]`, `meta.attempted`, `meta.quota` - **Task search**: `task`, `queries[]`, `grouped_results[]`, `merged_results[]`, `meta` - **Quota**: `date`, `providers[]`, `totals`; with `--real`: `real_quota.providers[]` --- ## Operator notes - Default mode: `workers=1` — conservative for cost control - Use `@dual` / `@deep` only for research tasks - `SearXNG` and `YaCy` are `enabled: false` by default (self-hosted only) - `MOJEEK_API_KEY` is optional — provider gracefully falls back to HTML scraping - Provider health data stored in `memory/provider-health/health.jsonl` - Discovery results stored in `memory/provider-discovery/discovery.jsonl` - Run `python -m free_search doctor` after setup to verify everything works - Run `python -m free_search discover` periodically to find new search sources

path: README.md
content: <p align="center">
  <img src="assets/hero.jpg?v=6d74a19" alt="Free Search Aggregator" width="100%" />
</p>

<h1 align="center">Free Search Aggregator</h1>

<p align="center">
Quota-aware web search for OpenClaw with 12 providers, automatic failover,
task-level multi-query search, and managed storage under <code>memory/</code>.
</p>

## What’s New (v1.2.x)

- Expanded provider pool to **12 providers**
- Added: **Exa, Bing RSS, Mojeek, Wikipedia, Google CSE, Baidu, SearXNG**
- Introduced clearer quota capability mapping for new providers
- Updated docs and setup guidance for API + no-key providers
- Smoke-tested core flow (`status/search/task/remaining/storage`) after rollout

---

## Highlights

- **12 providers** (6 work without API keys)
- **Auto failover** when provider fails (auth/rate/network/upstream)
- **Task search** (`task`) with query expansion + deduped merged results
- **Quota controls**: tracked quota + real quota (supported providers)
- **Preset modes**:
  - default: `workers=1`
  - `@dual`: `workers=2`
  - `@deep`: `workers=3` + deeper query coverage
- **Managed storage** (cache/index/report) to avoid workspace clutter

---

## Provider Matrix

| Provider | Key Required | Free Quota | Notes |
|---|---|---:|---|
| `brave` | `BRAVE_API_KEY` | 2000/day | Independent index, high quality |
| `exa` | `EXA_API_KEY` | ~33/day (1000/month) | Semantic search |
| `tavily` | `TAVILY_API_KEY` | 1000/day | AI-agent optimized |
| `duckduckgo` | No | ~500/day | Privacy-first fallback |
| `bing_html` | No | ~300/day | Bing RSS endpoint (stable XML) |
| `mojeek` | Optional (`MOJEEK_API_KEY`) | ~200/day | Independent index |
| `serper` | `SERPER_API_KEY` | 2500/day | Google-backed results |
| `searchapi` | `SEARCHAPI_API_KEY` | 100/month | Multi-engine API |
| `google_cse` | `GOOGLE_API_KEY` + `GOOGLE_CX` | 100/day | Official Google CSE |
| `baidu` | `BAIDU_API_KEY` | 200/day | Strong CN coverage |
| `wikipedia` | No | ~1000/day | Factual/encyclopedic |
| `searxng` | No (self-hosted) | depends | Requires private instance |

> With all keys configured, total tracked daily capacity is ~8400+.

---

## Quick Start

```bash
# Normal search
scripts/search "latest ai agent frameworks 2026" --max-results 5

# Task-level search
scripts/search task "@dual Compare Claude vs GPT-4 for code generation" --max-results 5

# Deep task search
scripts/search task "@deep autonomous driving safety 2026" --max-results 8 --max-queries 10

# Tracked quota status
scripts/status

# Real quota (supported providers)
scripts/remaining --real

# Optional probe for header-only providers (may consume quota)
scripts/remaining --real --probe

# Cleanup old cache files (default: 14 days)
scripts/gc --cache-days 14
```

---

## CLI

```bash
python -m free_search "<query>"
python -m free_search task "<task>" [--workers 1|2|3] [--max-queries N]
python -m free_search status [--real] [--probe]
python -m free_search remaining --real [--probe]
python -m free_search gc --cache-days 14 [--report-days 90]
```

---

## Python API

```python
from free_search import search, task_search, get_quota_status, get_real_quota

payload = search("latest LLM eval benchmark", max_results=5)

task_payload = task_search(
    "Compare Claude vs GPT-4 for code generation",
    max_results_per_query=5,
    max_queries=6,
    max_workers=2,
)

status = get_quota_status()
real = get_real_quota()
```

---

## Post-install Smoke Check (Recommended)

```bash
# 1) Config/provider load
scripts/status --compact

# 2) Basic search smoke test
scripts/search "openclaw" --max-results 3 --compact

# 3) Verify storage paths
ls -la /home/openclaw/.openclaw/workspace/memory/search-cache/ | tail -n 5
ls -la /home/openclaw/.openclaw/workspace/memory/search-index/
ls -la /home/openclaw/.openclaw/workspace/memory/search-reports/ | tail -n 5

# 4) Optional real quota check
scripts/remaining --real --compact
```

---

## Data Organization (Managed)

All artifacts are persisted under `memory/`:

- `memory/search-cache/YYYY-MM-DD/*.json` — raw cache
- `memory/search-index/search-index.jsonl` — append-only index
- `memory/search-reports/YYYY-MM-DD/*.md` — human-readable reports

Recommended retention:

- cache: **14 days**
- reports: **long-term**

---

## Real Quota Support

| Provider | Real quota support |
|---|---|
| Tavily | ✅ official usage endpoint |
| SearchAPI | ✅ official account endpoint |
| Brave | ⚠ via `--probe` header check |
| Serper | ❌ no public endpoint |
| DuckDuckGo / Bing / Mojeek / Wikipedia | N/A |
| Exa / Google CSE / Baidu | ❌ no public quota endpoint |

---

## Notes

- `@deep` optimizes for **coverage depth**, not brute-force burn.
- If quota usage gets high, concurrency may auto-downgrade.
- Keep `workers=1` for default workflows, enable `@dual/@deep` only when needed.


path: README.zh-CN.md
content: <p align="center">
  <img src="assets/hero.jpg?v=6d74a19" alt="Free Search Aggregator" width="100%" />
</p>

<h1 align="center">Free Search Aggregator</h1>

<p align="center">
12 源搜索聚合器：自动故障切换、配额感知、任务级多查询检索，
并将结果统一存入 <code>memory/</code>。6 个无需密钥即可使用的搜索源。
</p>

---

## 核心能力

- **12 个搜索提供商** — 6 个无需 API 密钥即用
- **自动 failover**：提供商失败 → 立即切换到下一个
- **配额管理**：本地每日追踪，80% 时发出警告
- **任务搜索**：自动扩展查询、分组结果、去重合并
- **并发预设**：`@dual`（2线程）、`@deep`（3线程+更深查询）
- **结果持久化**：缓存/索引/报告三层结构化存储

---

## 提供商一览

| 提供商 | 是否需密钥 | 免费额度 | 索引来源 | 说明 |
|--------|-----------|---------|---------|------|
| `brave` | BRAVE_API_KEY | 2000次/天 | Brave 独立索引 | 高质量，隐私友好 |
| `exa` | EXA_API_KEY | ~33次/天（1000/月） | 神经语义+网页 | 语义搜索，独特结果 |
| `tavily` | TAVILY_API_KEY | 1000次/天 | 网页（AI 优化） | 专为 AI agent 设计 |
| `duckduckgo` | 无 | ~500次/天 | Bing+自有 | 无密钥，隐私友好 |
| `bing_html` | 无 | ~300次/天 | 微软 Bing RSS | 无密钥，XML feed 稳定 |
| `mojeek` | 无（或MOJEEK_API_KEY） | 200次/天 | Mojeek 独立索引 | 非 Google/Bing 镜像 |
| `serper` | SERPER_API_KEY | 2500次/天 | Google | 免费额度最大 |
| `searchapi` | SEARCHAPI_API_KEY | 100次/月 | Google/Bing 多引擎 | 多引擎支持 |
| `google_cse` | GOOGLE_API_KEY + GOOGLE_CX | 100次/天 | Google 官方 | 官方 Google API |
| `baidu` | BAIDU_API_KEY | 200次/天 | 百度 | 中文内容效果最好 |
| `wikipedia` | 无 | 1000次/天 | Wikipedia | 事实性/百科查询 |
| `searxng` | 无 | 无限（自托管） | 元搜索聚合 | 需自托管实例 |

**全部密钥配置后：总配额 8400+ 次/天**

---

## 快速开始

```bash
# 普通搜索（无需任何配置即可使用）
scripts/search "米兰今日天气 2026" --max-results 5

# 任务搜索（多查询并行）
scripts/search task "@dual 对比 Claude 与 GPT-4 在代码生成上的差异" --max-results 5

# 深度研究模式
scripts/search task "@deep 自动驾驶安全技术 2026" --max-results 8 --max-queries 10

# 本地配额状态
scripts/status

# 真实配额（支持的提供商）
scripts/remaining --real

# 清理旧缓存
scripts/gc --cache-days 14
```

---

## 各提供商配置指南

### Bing RSS (`bing_html`) — 无需密钥
开箱即用。使用 Bing 内置 RSS 接口（`format=rss`），绕过反爬检测，返回稳定 XML 结果。

### Mojeek — 无需密钥（可选 API Key）
**开箱即用**（HTML 抓取模式）。如需更高稳定性和额度：
1. 注册：https://www.mojeek.com/services/search/api/
2. 设置 `MOJEEK_API_KEY` → 自动切换为官方 JSON API 模式

### Wikipedia — 无需密钥，支持多语言
修改 `providers.yaml` 中的 `lang` 配置切换语言：
```yaml
wikipedia:
  lang: zh   # en | zh | it | de | fr | ja ...
```

### Exa.ai — 语义搜索
1. 注册：https://exa.ai/
2. 设置 `EXA_API_KEY`
3. 免费额度：1000次/月（≈33次/天）
4. 支持 `auto`（自动）/ `neural`（语义）/ `keyword`（关键词）模式

### Google Custom Search
1. 申请 API Key：https://developers.google.com/custom-search/v1/introduction
2. 创建搜索引擎获取 CX：https://programmablesearchengine.google.com/
3. 设置 `GOOGLE_API_KEY` 和 `GOOGLE_CX`
4. 免费：100次/天

### 百度千帆 AI Search
1. 注册：https://cloud.baidu.com/
2. 设置 `BAIDU_API_KEY`
3. 中文内容检索效果最优

### SearXNG — 需自托管实例
公共实例对服务端请求限速严格，建议自托管：
```bash
docker run -d -p 8080:8080 searxng/searxng
```
然后在 `providers.yaml` 中：
```yaml
searxng:
  endpoint: http://localhost:8080
  enabled: true
```

---

## CLI 用法

```bash
python -m free_search "<query>"
python -m free_search task "<task>" [--workers 1|2|3] [--max-queries N]
python -m free_search status [--real] [--compact]
python -m free_search remaining --real [--probe]
python -m free_search gc --cache-days 14 [--report-days 90]
```

---

## Python API

```python
from free_search import search, task_search, get_quota_status, get_real_quota

# 普通搜索
payload = search("最新 LLM 评测", max_results=5)

# 任务搜索（多查询聚合）
task_payload = task_search(
    "对比 Claude 与 GPT-4 在代码生成上的差异",
    max_results_per_query=5,
    max_queries=6,
    max_workers=2,
)

# 配额状态
status = get_quota_status()
real = get_real_quota()
```

---

## 数据组织

所有搜索产物统一写入 `memory/`：

- `memory/search-cache/YYYY-MM-DD/*.json` — 原始结果缓存（14天）
- `memory/search-index/search-index.jsonl` — 追加式索引
- `memory/search-reports/YYYY-MM-DD/*.md` — 可读报告（长期保留）

---

## 真实配额支持情况

| 提供商 | 真实配额查询 | 说明 |
|--------|------------|------|
| `tavily` | ✅ 支持 | 官方 usage 接口 |
| `searchapi` | ✅ 支持 | 官方 account 接口 |
| `brave` | ⚠️ 探测（消耗一次请求） | 通过响应头读取 |
| `serper` | ❌ 不支持 | 无公开配额接口 |
| `duckduckgo/bing/mojeek/wikipedia` | — | 无配额概念 |
| `exa/google_cse/baidu` | ❌ 不支持 | 无公开配额接口 |

---

## 使用建议

- 默认 `workers=1`，仅在需要时使用 `@dual/@deep`
- `SearXNG` 和 `YaCy` 需要自托管，默认禁用
- `MOJEEK_API_KEY` 为可选项，不设置时自动使用 HTML 抓取
- 建议配置 `BRAVE_API_KEY` + `TAVILY_API_KEY` 获得最佳体验


path: agents/openai.yaml
content: interface:
  display_name: "Free Search Aggregator"
  short_description: "Self-healing failover web search with quality optimization and auto-discovery"
  default_prompt: "Search the web with automatic provider failover, health-based smart routing, result quality optimization, and auto-discovery of new search sources."


path: assets/hero.svg
content: <svg width="1600" height="640" viewBox="0 0 1600 640" xmlns="http://www.w3.org/2000/svg" fill="none">
  <defs>
    <linearGradient id="bg" x1="0" y1="0" x2="1600" y2="640" gradientUnits="userSpaceOnUse">
      <stop offset="0" stop-color="#F7F9FC"/>
      <stop offset="1" stop-color="#EEF3FA"/>
    </linearGradient>
    <linearGradient id="ink" x1="0" y1="0" x2="1" y2="1">
      <stop stop-color="#1E3A8A"/>
      <stop offset="1" stop-color="#334155"/>
    </linearGradient>
  </defs>

  <rect width="1600" height="640" rx="28" fill="url(#bg)"/>

  <!-- soft structure -->
  <g opacity="0.35" stroke="#CBD5E1" stroke-width="1">
    <path d="M120 142H1480"/>
    <path d="M120 498H1480"/>
  </g>

  <!-- emblem -->
  <g transform="translate(140,174)">
    <rect width="220" height="220" rx="24" fill="#FFFFFF" stroke="#D6E0EE"/>
    <circle cx="110" cy="110" r="10" fill="#0F172A"/>
    <circle cx="58" cy="58" r="8" fill="#2563EB"/>
    <circle cx="162" cy="58" r="8" fill="#2563EB"/>
    <circle cx="58" cy="162" r="8" fill="#334155"/>
    <circle cx="162" cy="162" r="8" fill="#334155"/>
    <path d="M110 110L58 58M110 110L162 58M110 110L58 162M110 110L162 162" stroke="url(#ink)" stroke-width="3" stroke-linecap="round"/>
  </g>

  <text x="430" y="280" fill="#0F172A" font-family="Inter,Segoe UI,Arial,sans-serif" font-size="70" font-weight="700" letter-spacing="-1.2">
    Free Search Aggregator
  </text>
  <text x="432" y="332" fill="#475569" font-family="Inter,Segoe UI,Arial,sans-serif" font-size="28" font-weight="500">
    Unified search orchestration with failover, quota control, and clean data management
  </text>

  <g transform="translate(432,384)">
    <rect x="0" y="0" width="170" height="42" rx="21" fill="#E2E8F0"/>
    <text x="85" y="27" text-anchor="middle" fill="#0F172A" font-family="Inter,Segoe UI,Arial,sans-serif" font-size="17" font-weight="600">Task Search</text>

    <rect x="184" y="0" width="180" height="42" rx="21" fill="#E2E8F0"/>
    <text x="274" y="27" text-anchor="middle" fill="#0F172A" font-family="Inter,Segoe UI,Arial,sans-serif" font-size="17" font-weight="600">Quota Aware</text>

    <rect x="378" y="0" width="190" height="42" rx="21" fill="#E2E8F0"/>
    <text x="473" y="27" text-anchor="middle" fill="#0F172A" font-family="Inter,Segoe UI,Arial,sans-serif" font-size="17" font-weight="600">Managed Storage</text>
  </g>
</svg>


path: config/providers.yaml
content: providers:
  # ── 高质量 API 搜索（需密钥，免费额度充足）─────────────────────────────
  brave:
    [REDACTED_SECRET]
    daily_quota: 2000
    enabled: true
    min_interval_seconds: 1.0
    timeout_seconds: 12

  exa:
    # Exa.ai 神经/语义搜索 — 免费 1000次/月 (≈33次/天)
    # 申请: https://exa.ai/  env: EXA_API_KEY
    [REDACTED_SECRET]
    daily_quota: 33
    enabled: true
    min_interval_seconds: 1.0
    timeout_seconds: 12
    search_type: auto      # auto | neural | keyword
    include_text: true

  tavily:
    [REDACTED_SECRET]
    daily_quota: 1000
    enabled: true
    min_interval_seconds: 1.0
    search_depth: basic
    timeout_seconds: 12

  # ── 完全免费（无需密钥）──────────────────────────────────────────────────
  duckduckgo:
    daily_quota: 500
    enabled: true
    min_interval_seconds: 1.0
    timeout_seconds: 12

  bing_html:
    # Bing RSS 搜索 — 无需密钥，使用 Bing 内置 RSS 接口（稳定无反爬）
    daily_quota: 300
    enabled: true
    min_interval_seconds: 2.0
    timeout_seconds: 15

  mojeek:
    # Mojeek — 独立索引，非 Google/Bing 镜像，无需密钥即可 HTML 抓取
    # 可选: 配置 MOJEEK_API_KEY 使用官方 JSON API 提升稳定性
    # 申请 API key: https://www.mojeek.com/services/search/api/
    [REDACTED_SECRET]
    daily_quota: 200
    enabled: true
    min_interval_seconds: 2.0
    timeout_seconds: 15

  wikipedia:
    # Wikipedia Search API — 完全免费，适合事实性/百科查询
    lang: en               # en | zh | ja | de | fr | it ...
    daily_quota: 1000
    enabled: true
    min_interval_seconds: 0.5
    timeout_seconds: 10

  duckduckgo_instant:
    daily_quota: 1000
    enabled: true
    min_interval_seconds: 0.5
    timeout_seconds: 12

  searxng:
    # SearXNG 元搜索 — 需自托管实例（公共实例服务端限速严格）
    # 自托管教程: https://searxng.github.io/searxng/
    # 配置好后修改 endpoint 并将 enabled 改为 true
    endpoint: ""           # 填入自托管实例地址，如: http://localhost:8080
    categories: general    # general | news | science
    daily_quota: 500
    enabled: false         # 默认禁用：需要自托管实例
    min_interval_seconds: 1.0
    timeout_seconds: 15

  # ── API 搜索（需密钥）────────────────────────────────────────────────────
  serper:
    [REDACTED_SECRET]
    daily_quota: 2500
    enabled: true
    min_interval_seconds: 1.0
    timeout_seconds: 12

  searchapi:
    [REDACTED_SECRET]
    daily_quota: 100
    enabled: true
    engine: google
    min_interval_seconds: 1.0
    timeout_seconds: 12

  google_cse:
    # Google Custom Search JSON API — 免费 100次/天
    # 1. 申请 API key: https://developers.google.com/custom-search/v1/introduction
    # 2. 创建搜索引擎获取 cx: https://programmablesearchengine.google.com/
    # env: GOOGLE_API_KEY, GOOGLE_CX
    [REDACTED_SECRET]
    cx: ${GOOGLE_CX}
    daily_quota: 100
    enabled: true
    min_interval_seconds: 1.0
    timeout_seconds: 12

  baidu:
    # 百度千帆 AI Search — 中文内容效果优异
    # 申请: https://cloud.baidu.com/  env: BAIDU_API_KEY
    [REDACTED_SECRET]
    daily_quota: 200
    enabled: true
    min_interval_seconds: 1.0
    timeout_seconds: 15
    edition: standard           # standard | enhanced
    search_recency_filter: year # year | month | week | day

  yacy:
    # YaCy 本地/P2P 搜索引擎（自托管）— 默认禁用
    endpoint: http://localhost:8090/yacysearch.json
    daily_quota: 1000
    enabled: false
    min_interval_seconds: 0.5
    timeout_seconds: 15
    resource: global

router:
  # 优先顺序: 高质量 API → 无密钥免费（Bing/Mojeek） → 其他 API → 百科保底
  # 密钥未设置的提供商会快速跳过（不消耗额度，不增加延迟）
  # SearXNG/YaCy 需自托管，默认禁用（在 providers 中 enabled: false）
  provider_order:
    - brave          # Brave 独立索引，高质量
    - exa            # Exa.ai 神经语义搜索
    - tavily         # 专为 AI agent 设计
    - duckduckgo     # 无密钥，隐私友好
    - bing_html      # 无密钥，Bing RSS，微软索引
    - mojeek         # 无密钥，独立索引（非 Google/Bing）
    - serper         # Google 索引，2500次/天
    - searchapi      # Google/Bing 多引擎
    - google_cse     # Google 官方 API，100次/天
    - baidu          # 百度，中文内容
    - wikipedia      # 百科事实查询保底
    - searxng        # 自托管 SearXNG（禁用中）
  quota_state_file: .quota_state.json


path: requirements.txt
content: beautifulsoup4>=4.12.3
PyYAML>=6.0.2
requests>=2.32.3


path: scripts/fsearch.sh
content: #!/bin/bash
# Free Search Aggregator - 默认搜索入口
# 用法: fsearch "查询内容"

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"

export PYTHONPATH="${ROOT_DIR}/src${PYTHONPATH:+:${PYTHONPATH}}"

# 从环境变量读取 API Keys
export BRAVE_API_KEY="${BRAVE_API_KEY:-}"
export TAVILY_API_KEY="${TAVILY_API_KEY:-}"
export SERPER_API_KEY="${SERPER_API_KEY:-}"
export SEARCHAPI_API_KEY="${SEARCHAPI_API_KEY:-}"

exec python3 -m free_search "$@"


path: src/free_search/__init__.py
content: """Free Search Aggregator public API.

Usage:
    from free_search import search
    data = search("latest ai benchmarks", max_results=5)
"""

from __future__ import annotations

import logging
from pathlib import Path
from typing import Any

from .discovery import SourceDiscovery
from .health import HealthTracker
from .router import SearchRouter, SearchRouterError
from .task_search import task_search
from .quota_real import get_real_quota


def configure_logging(level: str = "INFO") -> None:
    """Configure package-level logging for consumers that do not set handlers."""
    numeric = getattr(logging, level.upper(), logging.INFO)
    logging.basicConfig(
        level=numeric,
        format="%(asctime)s %(levelname)s %(name)s :: %(message)s",
    )


def search(
    query: str,
    *,
    max_results: int = 8,
    config_path: str | None = None,
) -> dict[str, Any]:
    """Run a search with automatic provider failover.

    Args:
        query: Search query.
        max_results: Maximum results to return.
        config_path: Optional explicit path to providers.yaml.

    Returns:
        Dictionary with unified `results` and `meta` fields.
    """
    if not query or not query.strip():
        raise ValueError("query must be non-empty")
    if max_results <= 0:
        raise ValueError("max_results must be > 0")

    router = SearchRouter(config_path=config_path)
    return router.search(query.strip(), max_results=max_results)

def get_quota_status(*, config_path: str | None = None) -> dict[str, Any]:
    """Return per-provider daily quota usage without running a search."""
    router = SearchRouter(config_path=config_path)
    return router.quota_status()


def reset_quota(*, config_path: str | None = None, provider: str | None = None) -> dict[str, Any]:
    """Reset daily quota usage (all providers or a single provider)."""
    router = SearchRouter(config_path=config_path)
    return router.reset_quota(provider=provider)


def get_health_status(*, window_hours: int = 72) -> dict[str, Any]:
    """Return provider health summary."""
    tracker = HealthTracker()
    return tracker.get_summary(window_hours=window_hours)


def run_discovery(*, config_path: str | None = None) -> dict[str, Any]:
    """Run full source discovery cycle."""
    discovery = SourceDiscovery(config_path=config_path)
    return discovery.run_discovery()


__all__ = [
    "search",
    "task_search",
    "get_real_quota",
    "get_quota_status",
    "reset_quota",
    "get_health_status",
    "run_discovery",
    "configure_logging",
    "SearchRouter",
    "SearchRouterError",
    "HealthTracker",
    "SourceDiscovery",
]


def _default_config_path() -> Path:
    return Path(__file__).resolve().parents[2] / "config" / "providers.yaml"


path: src/free_search/__main__.py
content: """CLI entrypoint for subprocess usage: python -m free_search."""

from __future__ import annotations

import argparse
import json
import sys
from typing import Any

from . import configure_logging, get_quota_status, reset_quota, search, task_search, get_real_quota
from .discovery import SourceDiscovery
from .health import HealthTracker
from .router import SearchRouter, SearchRouterError
from .storage import persist_search_payload, run_gc


def _normalize_compat_tokens(argv: list[str]) -> list[str]:
    if len(argv) >= 3 and argv[0].lower() == "brave" and argv[1].lower() == "search":
        return argv[2:]
    if len(argv) >= 2 and argv[0].lower() == "search":
        return argv[1:]
    return argv


_ENV_KEY_MAP = {
    "brave": ["BRAVE_API_KEY"],
    "exa": ["EXA_API_KEY"],
    "tavily": ["TAVILY_API_KEY"],
    "serper": ["SERPER_API_KEY"],
    "searchapi": ["SEARCHAPI_API_KEY"],
    "google_cse": ["GOOGLE_API_KEY", "GOOGLE_CX"],
    "baidu": ["BAIDU_API_KEY"],
    "mojeek": ["MOJEEK_API_KEY"],
}

_FREE_PROVIDERS = {"duckduckgo", "bing_html", "wikipedia", "duckduckgo_instant"}


def _run_doctor(*, config_path: str | None = None) -> dict[str, Any]:
    """Run system diagnostics: config, providers, storage, health."""
    import os
    from .storage import _memory_root

    checks: list[dict[str, Any]] = []

    # 1. Config file check
    try:
        router = SearchRouter(config_path=config_path)
        checks.append({"check": "config_load", "status": "ok", "providers_configured": len(router.order)})
    except Exception as exc:
        checks.append({"check": "config_load", "status": "error", "detail": str(exc)})
        return {"checks": checks, "overall": "error"}

    # 2. Provider credential check
    provider_status: list[dict[str, Any]] = []
    for name in router.order:
        provider = router.providers.get(name)
        if not provider:
            provider_status.append({"provider": name, "status": "not_loaded"})
            continue
        if not provider.is_enabled():
            provider_status.append({"provider": name, "status": "disabled"})
            continue

        # Check if API key is set for key-required providers
        if name in _ENV_KEY_MAP:
            missing = [k for k in _ENV_KEY_MAP[name] if not os.environ.get(k)]
            if missing:
                provider_status.append({"provider": name, "status": "missing_keys", "keys": missing})
                continue

        provider_status.append({"provider": name, "status": "ready"})

    checks.append({"check": "providers", "status": "ok", "providers": provider_status})

    # 3. Storage directory check
    try:
        mem_root = _memory_root()
        mem_root.mkdir(parents=True, exist_ok=True)
        writable = os.access(str(mem_root), os.W_OK)
        checks.append({"check": "storage", "status": "ok" if writable else "warning",
                       "path": str(mem_root), "writable": writable})
    except Exception as exc:
        checks.append({"check": "storage", "status": "error", "detail": str(exc)})

    # 4. Health data check
    try:
        tracker = HealthTracker()
        summary = tracker.get_summary(window_hours=24)
        checks.append({"check": "health_data", "status": "ok",
                       "records_24h": summary.get("total_records", 0)})
    except Exception as exc:
        checks.append({"check": "health_data", "status": "error", "detail": str(exc)})

    # 5. Quota check
    try:
        quota = get_quota_status(config_path=config_path)
        high_usage = [
            p["provider"] for p in quota.get("providers", [])
            if isinstance(p.get("percentage_used"), (int, float)) and p["percentage_used"] >= 80
        ]
        checks.append({"check": "quota", "status": "warning" if high_usage else "ok",
                       "high_usage_providers": high_usage})
    except Exception as exc:
        checks.append({"check": "quota", "status": "error", "detail": str(exc)})

    # Overall status
    statuses = [c["status"] for c in checks]
    if "error" in statuses:
        overall = "error"
    elif "warning" in statuses:
        overall = "warning"
    else:
        overall = "ok"

    ready_count = sum(1 for p in provider_status if p["status"] == "ready")

    return {
        "overall": overall,
        "ready_providers": ready_count,
        "total_providers": len(provider_status),
        "checks": checks,
    }


def _run_setup(*, config_path: str | None = None) -> dict[str, Any]:
    """Show setup status and configuration recommendations."""
    import os

    providers_info: list[dict[str, Any]] = []

    # Free providers (always available)
    for name in sorted(_FREE_PROVIDERS):
        providers_info.append({
            "provider": name,
            "type": "free",
            "status": "ready",
            "action": "No setup required",
        })

    # API-key providers
    for name, keys in sorted(_ENV_KEY_MAP.items()):
        set_keys = [k for k in keys if os.environ.get(k)]
        missing_keys = [k for k in keys if not os.environ.get(k)]

        if not missing_keys:
            providers_info.append({
                "provider": name,
                "type": "api_key",
                "status": "ready",
                "action": "Configured and ready",
            })
        else:
            providers_info.append({
                "provider": name,
                "type": "api_key",
                "status": "needs_setup",
                "missing_keys": missing_keys,
                "action": f"Set environment variable(s): {', '.join(missing_keys)}",
            })

    ready = sum(1 for p in providers_info if p["status"] == "ready")
    needs_setup = sum(1 for p in providers_info if p["status"] == "needs_setup")

    recommendations: list[str] = []
    if needs_setup > 0:
        recommendations.append(f"{needs_setup} providers need API keys — set the environment variables to enable them")
    if ready < 4:
        recommendations.append("Consider setting up more API-key providers for better failover coverage")
    recommendations.append("Run 'python -m free_search doctor' for a full system diagnostic")
    recommendations.append("Run 'python -m free_search discover' to find additional search sources")

    return {
        "ready_providers": ready,
        "needs_setup": needs_setup,
        "providers": providers_info,
        "recommendations": recommendations,
    }


def main(argv: list[str] | None = None) -> int:
    raw_args = list(argv) if argv is not None else sys.argv[1:]
    normalized_args = _normalize_compat_tokens(raw_args)

    if normalized_args and normalized_args[0].lower() in ("status", "remaining", "quota"):
        parser = argparse.ArgumentParser(description="Show or reset provider quota usage")
        parser.add_argument("--config", default=None, help="Path to providers.yaml")
        parser.add_argument("--log-level", default="INFO")
        parser.add_argument("--compact", action="store_true", help="Print compact JSON")
        parser.add_argument("--reset", action="store_true", help="Reset quota counters before showing status")
        parser.add_argument("--real", action="store_true", help="Fetch real provider quota when supported")
        parser.add_argument("--probe", action="store_true", help="Allow probe requests for providers without quota endpoints (may consume quota)")
        args = parser.parse_args(normalized_args[1:])

        configure_logging(args.log_level)
        try:
            payload = (
                reset_quota(config_path=args.config)
                if args.reset
                else get_quota_status(config_path=args.config)
            )
        except Exception as exc:  # pragma: no cover - defensive for CLI consumers
            print(json.dumps({"error": f"unexpected_error: {exc}"}, ensure_ascii=False), file=sys.stderr)
            return 1

        # Add convenience totals for quick inspection
        totals = {"total_remaining": 0, "total_daily_quota": 0}
        for p in payload.get("providers", []):
            dq = p.get("daily_quota")
            rem = p.get("remaining")
            if isinstance(dq, int):
                totals["total_daily_quota"] += dq
            if isinstance(rem, int):
                totals["total_remaining"] += rem
        payload["totals"] = totals

        if args.real:
            real = get_real_quota(config_path=args.config, probe_brave=bool(args.probe))
            payload["real_quota"] = real

        indent = None if args.compact else 2
        print(json.dumps(payload, ensure_ascii=False, indent=indent))
        return 0

    if normalized_args and normalized_args[0].lower() == "gc":
        parser = argparse.ArgumentParser(description="Clean old search cache/report files")
        parser.add_argument("--cache-days", type=int, default=14)
        parser.add_argument("--report-days", type=int, default=None)
        parser.add_argument("--compact", action="store_true", help="Print compact JSON")
        args = parser.parse_args(normalized_args[1:])

        payload = run_gc(cache_days=args.cache_days, report_days=args.report_days)
        indent = None if args.compact else 2
        print(json.dumps(payload, ensure_ascii=False, indent=indent))
        return 0

    if normalized_args and normalized_args[0].lower() == "task":
        parser = argparse.ArgumentParser(description="Run task-level multi-query search")
        parser.add_argument("task", nargs="+", help="Task or goal statement")
        parser.add_argument("--max-results", type=int, default=5)
        parser.add_argument("--max-queries", type=int, default=6)
        parser.add_argument("--max-merged-results", type=int, default=None)
        parser.add_argument("--workers", type=int, default=1)
        parser.add_argument("--config", default=None, help="Path to providers.yaml")
        parser.add_argument("--log-level", default="INFO")
        parser.add_argument("--compact", action="store_true", help="Print compact JSON")
        args = parser.parse_args(normalized_args[1:])
        task = " ".join(args.task).strip()

        # Preset prefixes for interaction convenience:
        # - @dual: workers=2
        # - @deep: workers=3 and max_queries>=8 (depth first, not brute force concurrency)
        effective_workers = args.workers
        effective_max_queries = args.max_queries
        if task.startswith("@dual"):
            task = task[len("@dual"):].strip()
            effective_workers = 2
        elif task.startswith("@deep"):
            task = task[len("@deep"):].strip()
            effective_workers = 3
            effective_max_queries = max(effective_max_queries, 8)

        # Quota-aware degradation: when quota usage is high, reduce concurrency.
        try:
            status = get_quota_status(config_path=args.config)
            max_pct = 0.0
            for p in status.get("providers", []):
                pct = p.get("percentage_used")
                if isinstance(pct, (int, float)):
                    max_pct = max(max_pct, float(pct))
            if max_pct >= 80.0 and effective_workers > 2:
                effective_workers = 2
        except Exception:
            pass

        configure_logging(args.log_level)
        try:
            payload = task_search(
                task,
                max_results_per_query=args.max_results,
                max_queries=effective_max_queries,
                max_merged_results=args.max_merged_results,
                max_workers=effective_workers,
                config_path=args.config,
            )
        except SearchRouterError as exc:
            print(json.dumps({"error": str(exc)}, ensure_ascii=False), file=sys.stderr)
            return 2
        except Exception as exc:  # pragma: no cover - defensive for CLI consumers
            print(json.dumps({"error": f"unexpected_error: {exc}"}, ensure_ascii=False), file=sys.stderr)
            return 1

        storage_info = persist_search_payload(query=task, payload=payload, mode="task")
        payload.setdefault("meta", {})["storage"] = storage_info

        indent = None if args.compact else 2
        print(json.dumps(payload, ensure_ascii=False, indent=indent))
        return 0

    # ── health command ──────────────────────────────────────────────────────
    if normalized_args and normalized_args[0].lower() == "health":
        parser = argparse.ArgumentParser(description="Show provider health dashboard")
        parser.add_argument("--window", type=int, default=72, help="Analysis window in hours")
        parser.add_argument("--compact", action="store_true", help="Print compact JSON")
        parser.add_argument("--compact-health", action="store_true", help="Compact old health records")
        args = parser.parse_args(normalized_args[1:])

        tracker = HealthTracker()

        if args.compact_health:
            removed = tracker.compact(keep_hours=args.window)
            print(json.dumps({"compacted": True, "records_removed": removed}, indent=2))
            return 0

        payload = tracker.get_summary(window_hours=args.window)
        indent = None if args.compact else 2
        print(json.dumps(payload, ensure_ascii=False, indent=indent))
        return 0

    # ── discover command ─────────────────────────────────────────────────────
    if normalized_args and normalized_args[0].lower() == "discover":
        parser = argparse.ArgumentParser(description="Discover and probe search sources")
        parser.add_argument("--config", default=None, help="Path to providers.yaml")
        parser.add_argument("--compact", action="store_true", help="Print compact JSON")
        parser.add_argument("--log-level", default="INFO")
        args = parser.parse_args(normalized_args[1:])

        configure_logging(args.log_level)
        discovery = SourceDiscovery(config_path=args.config)
        payload = discovery.run_discovery()

        indent = None if args.compact else 2
        print(json.dumps(payload, ensure_ascii=False, indent=indent))
        return 0

    # ── doctor command ───────────────────────────────────────────────────────
    if normalized_args and normalized_args[0].lower() == "doctor":
        parser = argparse.ArgumentParser(description="Run system diagnostics")
        parser.add_argument("--config", default=None, help="Path to providers.yaml")
        parser.add_argument("--compact", action="store_true", help="Print compact JSON")
        parser.add_argument("--log-level", default="WARNING")
        args = parser.parse_args(normalized_args[1:])

        configure_logging(args.log_level)
        diagnostics = _run_doctor(config_path=args.config)

        indent = None if args.compact else 2
        print(json.dumps(diagnostics, ensure_ascii=False, indent=indent))
        return 0

    # ── setup command ────────────────────────────────────────────────────────
    if normalized_args and normalized_args[0].lower() == "setup":
        parser = argparse.ArgumentParser(description="Show setup status and recommendations")
        parser.add_argument("--config", default=None, help="Path to providers.yaml")
        parser.add_argument("--compact", action="store_true", help="Print compact JSON")
        args = parser.parse_args(normalized_args[1:])

        payload = _run_setup(config_path=args.config)

        indent = None if args.compact else 2
        print(json.dumps(payload, ensure_ascii=False, indent=indent))
        return 0

    # ── default: search ──────────────────────────────────────────────────────
    parser = argparse.ArgumentParser(description="Run failover web search")
    parser.add_argument("query", nargs="+", help="Search query")
    parser.add_argument("--max-results", type=int, default=8)
    parser.add_argument("--config", default=None, help="Path to providers.yaml")
    parser.add_argument("--log-level", default="INFO")
    parser.add_argument("--compact", action="store_true", help="Print compact JSON")
    args = parser.parse_args(normalized_args)
    query = " ".join(args.query).strip()

    configure_logging(args.log_level)
    try:
        payload = search(query, max_results=args.max_results, config_path=args.config)
    except SearchRouterError as exc:
        print(json.dumps({"error": str(exc)}, ensure_ascii=False), file=sys.stderr)
        return 2
    except Exception as exc:  # pragma: no cover - defensive for CLI consumers
        print(json.dumps({"error": f"unexpected_error: {exc}"}, ensure_ascii=False), file=sys.stderr)
        return 1

    storage_info = persist_search_payload(query=query, payload=payload, mode="search")
    payload.setdefault("meta", {})["storage"] = storage_info

    indent = None if args.compact else 2
    print(json.dumps(payload, ensure_ascii=False, indent=indent))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: src/free_search/discovery.py
content: """Source discovery: probe providers, scan candidate sources, find new search engines."""

from __future__ import annotations

import json
import logging
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import UTC, datetime
from pathlib import Path
from typing import Any

import requests

from .health import HealthTracker
from .router import SearchRouter
from .storage import _memory_root

logger = logging.getLogger(__name__)

_PROBE_QUERY = "test"
_PROBE_TIMEOUT = 8
_PROBE_MAX_RESULTS = 3
_MAX_ACCEPTABLE_LATENCY_MS = 5000

# ── Candidate Search Sources ────────────────────────────────────────────────
# Free search engines / APIs that are not yet integrated as full providers
# but can be probed for availability and quality.

CANDIDATE_SOURCES: list[dict[str, Any]] = [
    {
        "name": "marginalia",
        "probe_url": "https://api.marginalia.nu/{subpath}",
        "search_url": "https://search.marginalia.nu/search?query={query}&count={count}&format=json",
        "type": "json_api",
        "description": "Independent index focused on non-commercial/small websites",
        "result_path": "results",
        "title_key": "title",
        "url_key": "url",
        "snippet_key": "description",
    },
    {
        "name": "wiby",
        "search_url": "https://wiby.me/json/?q={query}",
        "type": "json_api",
        "description": "Minimalist search engine for classic/Web 1.0 content",
        "result_path": None,  # root is array
        "title_key": "Title",
        "url_key": "URL",
        "snippet_key": "Snippet",
    },
    {
        "name": "curlie",
        "search_url": "https://curlie.org/search?q={query}&format=json",
        "type": "json_api",
        "description": "DMOZ successor, human-curated web directory",
        "result_path": "results",
        "title_key": "title",
        "url_key": "url",
        "snippet_key": "description",
    },
    # ── SearXNG Public Instances ──────────────────────────────────────────
    {
        "name": "searxng_paulgo",
        "search_url": "https://paulgo.io/search?q={query}&format=json",
        "type": "searxng",
        "description": "SearXNG public instance (paulgo.io)",
    },
    {
        "name": "searxng_priv",
        "search_url": "https://priv.au/search?q={query}&format=json",
        "type": "searxng",
        "description": "SearXNG public instance (priv.au)",
    },
    {
        "name": "searxng_northboot",
        "search_url": "https://search.bus-hit.me/search?q={query}&format=json",
        "type": "searxng",
        "description": "SearXNG public instance (bus-hit.me)",
    },
    {
        "name": "searxng_ononoki",
        "search_url": "https://search.ononoki.org/search?q={query}&format=json",
        "type": "searxng",
        "description": "SearXNG public instance (ononoki.org)",
    },
]


def _discovery_storage_path() -> Path:
    return _memory_root() / "provider-discovery" / "discovery.jsonl"


def _probe_existing_provider(
    router: SearchRouter,
    name: str,
) -> dict[str, Any]:
    """Probe a single configured provider with a test query."""
    provider = router.providers.get(name)
    if not provider:
        return {
            "provider": name,
            "status": "not_configured",
            "latency_ms": 0,
            "result_count": 0,
        }
    if not provider.is_enabled():
        return {
            "provider": name,
            "status": "disabled",
            "latency_ms": 0,
            "result_count": 0,
        }

    started = time.perf_counter()
    try:
        items = provider.search(_PROBE_QUERY, max_results=_PROBE_MAX_RESULTS)
        elapsed_ms = int((time.perf_counter() - started) * 1000)
        return {
            "provider": name,
            "status": "ok" if items else "empty",
            "latency_ms": elapsed_ms,
            "result_count": len(items) if items else 0,
        }
    except Exception as exc:
        elapsed_ms = int((time.perf_counter() - started) * 1000)
        return {
            "provider": name,
            "status": "error",
            "latency_ms": elapsed_ms,
            "result_count": 0,
            "error": f"{type(exc).__name__}: {exc}",
        }


def _probe_candidate_source(candidate: dict[str, Any]) -> dict[str, Any]:
    """Probe a candidate search source for availability and quality."""
    name = candidate["name"]
    search_url = candidate.get("search_url", "")
    source_type = candidate.get("type", "json_api")

    if not search_url:
        return {
            "name": name,
            "status": "no_url",
            "latency_ms": 0,
            "result_count": 0,
            "quality_score": 0.0,
        }

    url = search_url.replace("{query}", _PROBE_QUERY).replace("{count}", "3")

    started = time.perf_counter()
    try:
        headers = {
            "User-Agent": "Mozilla/5.0 (compatible; FreeSearchAggregator/1.0)",
            "Accept": "application/json",
        }
        resp = requests.get(url, headers=headers, timeout=_PROBE_TIMEOUT)
        elapsed_ms = int((time.perf_counter() - started) * 1000)

        if resp.status_code != 200:
            return {
                "name": name,
                "status": "http_error",
                "http_status": resp.status_code,
                "latency_ms": elapsed_ms,
                "result_count": 0,
                "quality_score": 0.0,
            }

        # Try to parse response
        result_count = 0
        quality_score = 0.0

        try:
            data = resp.json()

            if source_type == "searxng":
                results = data.get("results", []) if isinstance(data, dict) else []
                result_count = len(results)
            elif source_type == "json_api":
                result_path = candidate.get("result_path")
                if result_path is None:
                    # Root is array
                    results = data if isinstance(data, list) else []
                else:
                    results = data.get(result_path, []) if isinstance(data, dict) else []
                result_count = len(results) if isinstance(results, list) else 0
            else:
                result_count = 0

            # Quality scoring: result count + reasonable latency
            if result_count > 0:
                count_score = min(1.0, result_count / 5.0)
                latency_score = max(0.0, 1.0 - elapsed_ms / _MAX_ACCEPTABLE_LATENCY_MS)
                quality_score = round(count_score * 0.6 + latency_score * 0.4, 4)

        except (ValueError, KeyError):
            return {
                "name": name,
                "status": "parse_error",
                "latency_ms": elapsed_ms,
                "result_count": 0,
                "quality_score": 0.0,
            }

        status = "available" if result_count > 0 else "empty"
        return {
            "name": name,
            "type": source_type,
            "status": status,
            "latency_ms": elapsed_ms,
            "result_count": result_count,
            "quality_score": quality_score,
            "description": candidate.get("description", ""),
        }

    except requests.exceptions.Timeout:
        elapsed_ms = int((time.perf_counter() - started) * 1000)
        return {
            "name": name,
            "status": "timeout",
            "latency_ms": elapsed_ms,
            "result_count": 0,
            "quality_score": 0.0,
        }
    except requests.exceptions.RequestException as exc:
        elapsed_ms = int((time.perf_counter() - started) * 1000)
        return {
            "name": name,
            "status": "network_error",
            "latency_ms": elapsed_ms,
            "result_count": 0,
            "quality_score": 0.0,
            "error": str(exc),
        }


class SourceDiscovery:
    """Discovers and probes search sources — both configured and candidate."""

    def __init__(self, config_path: str | None = None) -> None:
        self.config_path = config_path
        self.storage_path = _discovery_storage_path()
        self.storage_path.parent.mkdir(parents=True, exist_ok=True)

    def probe_all(self) -> dict[str, Any]:
        """Probe all configured providers in parallel."""
        router = SearchRouter(config_path=self.config_path)
        results: list[dict[str, Any]] = []

        with ThreadPoolExecutor(max_workers=3) as pool:
            futs = {
                pool.submit(_probe_existing_provider, router, name): name
                for name in router.order
            }
            for fut in as_completed(futs):
                results.append(fut.result())

        # Sort by status: ok first, then by latency
        results.sort(key=lambda x: (0 if x["status"] == "ok" else 1, x.get("latency_ms", 9999)))
        return {
            "configured_providers": results,
            "timestamp_utc": datetime.now(UTC).isoformat(),
        }

    def scan_candidate_sources(self) -> list[dict[str, Any]]:
        """Probe all candidate sources in parallel."""
        results: list[dict[str, Any]] = []

        with ThreadPoolExecutor(max_workers=4) as pool:
            futs = {
                pool.submit(_probe_candidate_source, src): src["name"]
                for src in CANDIDATE_SOURCES
            }
            for fut in as_completed(futs):
                results.append(fut.result())

        # Sort: available first, then by quality score descending
        results.sort(key=lambda x: (
            0 if x["status"] == "available" else 1,
            -x.get("quality_score", 0),
        ))
        return results

    def run_discovery(self) -> dict[str, Any]:
        """Full discovery cycle: probe existing + scan candidates + generate recommendations."""
        logger.info("Starting source discovery...")

        # Phase 1: Probe existing providers
        existing = self.probe_all()

        # Phase 2: Scan candidate sources
        candidates = self.scan_candidate_sources()

        # Phase 3: Sync to health tracker
        health = HealthTracker()
        for p in existing.get("configured_providers", []):
            name = p.get("provider", "")
            if not name:
                continue
            health.record(
                name,
                success=p["status"] == "ok",
                latency_ms=p.get("latency_ms", 0),
                error_type=p.get("error", None) if p["status"] not in ("ok", "disabled", "not_configured") else None,
            )

        # Phase 4: Generate recommendations
        recommendations: list[str] = []
        for c in candidates:
            if c["status"] == "available" and c.get("quality_score", 0) >= 0.3:
                desc = c.get("description", c["name"])
                recommendations.append(
                    f"{c['name']} (quality={c['quality_score']:.2f}, latency={c['latency_ms']}ms): {desc}"
                )

        # Identify degraded existing providers
        degraded: list[str] = []
        for p in existing.get("configured_providers", []):
            if p["status"] in ("error", "empty") and p["provider"] not in ("searxng", "yacy"):
                degraded.append(f"{p['provider']}: {p.get('error', p['status'])}")

        result = {
            "existing_providers": existing.get("configured_providers", []),
            "candidate_results": candidates,
            "recommendations": recommendations,
            "degraded_providers": degraded,
            "timestamp_utc": datetime.now(UTC).isoformat(),
        }

        # Persist discovery result
        self._persist(result)

        return result

    def _persist(self, result: dict[str, Any]) -> None:
        """Append discovery result to storage."""
        try:
            with self.storage_path.open("a", encoding="utf-8") as f:
                entry = {
                    "ts": result.get("timestamp_utc", datetime.now(UTC).isoformat()),
                    "existing_ok": sum(
                        1 for p in result.get("existing_providers", [])
                        if p.get("status") == "ok"
                    ),
                    "existing_total": len(result.get("existing_providers", [])),
                    "candidates_available": sum(
                        1 for c in result.get("candidate_results", [])
                        if c.get("status") == "available"
                    ),
                    "candidates_total": len(result.get("candidate_results", [])),
                    "recommendations_count": len(result.get("recommendations", [])),
                    "degraded_count": len(result.get("degraded_providers", [])),
                }
                f.write(json.dumps(entry, ensure_ascii=False) + "\n")
        except OSError:
            logger.warning("Failed to persist discovery results", exc_info=True)


path: src/free_search/health.py
content: """Provider health tracking: record outcomes, compute scores, smart ordering."""

from __future__ import annotations

import json
import logging
import time
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import Any

from .storage import _memory_root

logger = logging.getLogger(__name__)

# Health score weights
_W_SUCCESS = 0.50
_W_LATENCY = 0.30
_W_FRESHNESS = 0.20

# Latency thresholds (ms) for scoring: <= fast → 1.0, >= slow → 0.0
_LATENCY_FAST_MS = 500
_LATENCY_SLOW_MS = 5000

# Default analysis window
_DEFAULT_WINDOW_HOURS = 72

# Minimum records needed before health-based reordering kicks in
_MIN_RECORDS_FOR_REORDER = 5


def _health_storage_path() -> Path:
    return _memory_root() / "provider-health" / "health.jsonl"


class HealthTracker:
    """Tracks provider performance and computes health scores."""

    def __init__(self, storage_path: Path | None = None) -> None:
        self.path = storage_path or _health_storage_path()
        self.path.parent.mkdir(parents=True, exist_ok=True)

    def record(
        self,
        provider: str,
        *,
        success: bool,
        latency_ms: int,
        error_type: str | None = None,
    ) -> None:
        """Append a health record for a provider attempt."""
        entry = {
            "ts": datetime.now(UTC).isoformat(),
            "provider": provider,
            "success": success,
            "latency_ms": latency_ms,
            "error_type": error_type,
        }
        try:
            with self.path.open("a", encoding="utf-8") as f:
                f.write(json.dumps(entry, ensure_ascii=False) + "\n")
        except OSError:
            logger.warning("Failed to write health record for %s", provider, exc_info=True)

    def _load_records(self, window_hours: int = _DEFAULT_WINDOW_HOURS) -> list[dict[str, Any]]:
        """Load records within the time window."""
        if not self.path.exists():
            return []

        cutoff = datetime.now(UTC) - timedelta(hours=window_hours)
        records: list[dict[str, Any]] = []

        try:
            with self.path.open("r", encoding="utf-8") as f:
                for line in f:
                    line = line.strip()
                    if not line:
                        continue
                    try:
                        rec = json.loads(line)
                        ts = datetime.fromisoformat(rec["ts"])
                        if ts >= cutoff:
                            records.append(rec)
                    except (json.JSONDecodeError, KeyError, ValueError):
                        continue
        except OSError:
            logger.warning("Failed to read health records", exc_info=True)

        return records

    def get_scores(self, window_hours: int = _DEFAULT_WINDOW_HOURS) -> dict[str, float]:
        """Compute health scores (0.0–1.0) per provider within time window."""
        records = self._load_records(window_hours)
        if not records:
            return {}

        # Group by provider
        by_provider: dict[str, list[dict[str, Any]]] = {}
        for rec in records:
            name = rec.get("provider", "")
            if name:
                by_provider.setdefault(name, []).append(rec)

        now = datetime.now(UTC)
        scores: dict[str, float] = {}

        for name, recs in by_provider.items():
            # Success rate
            total = len(recs)
            successes = sum(1 for r in recs if r.get("success"))
            success_rate = successes / total if total > 0 else 0.0

            # Average latency score
            latencies = [r.get("latency_ms", _LATENCY_SLOW_MS) for r in recs]
            avg_latency = sum(latencies) / len(latencies) if latencies else _LATENCY_SLOW_MS
            # Map latency to 0.0–1.0 (lower is better)
            if avg_latency <= _LATENCY_FAST_MS:
                latency_score = 1.0
            elif avg_latency >= _LATENCY_SLOW_MS:
                latency_score = 0.0
            else:
                latency_score = 1.0 - (avg_latency - _LATENCY_FAST_MS) / (_LATENCY_SLOW_MS - _LATENCY_FAST_MS)

            # Freshness: how recent is the latest record
            latest_ts = max(
                (datetime.fromisoformat(r["ts"]) for r in recs),
                default=now - timedelta(hours=window_hours),
            )
            age_hours = (now - latest_ts).total_seconds() / 3600.0
            freshness = max(0.0, 1.0 - age_hours / window_hours)

            score = (
                _W_SUCCESS * success_rate
                + _W_LATENCY * latency_score
                + _W_FRESHNESS * freshness
            )
            scores[name] = round(min(1.0, max(0.0, score)), 4)

        return scores

    def get_summary(self, window_hours: int = _DEFAULT_WINDOW_HOURS) -> dict[str, Any]:
        """Return a dashboard-friendly summary of provider health."""
        records = self._load_records(window_hours)
        scores = self.get_scores(window_hours)

        # Group stats
        by_provider: dict[str, dict[str, Any]] = {}
        for rec in records:
            name = rec.get("provider", "")
            if not name:
                continue
            stats = by_provider.setdefault(name, {
                "total": 0, "successes": 0, "failures": 0,
                "latencies": [], "error_types": {},
            })
            stats["total"] += 1
            if rec.get("success"):
                stats["successes"] += 1
            else:
                stats["failures"] += 1
                et = rec.get("error_type", "unknown")
                stats["error_types"][et] = stats["error_types"].get(et, 0) + 1
            stats["latencies"].append(rec.get("latency_ms", 0))

        providers: list[dict[str, Any]] = []
        for name in sorted(by_provider.keys()):
            s = by_provider[name]
            lats = s["latencies"]
            providers.append({
                "provider": name,
                "health_score": scores.get(name, 0.0),
                "total_requests": s["total"],
                "success_rate": round(s["successes"] / s["total"], 4) if s["total"] > 0 else 0.0,
                "avg_latency_ms": round(sum(lats) / len(lats)) if lats else 0,
                "min_latency_ms": min(lats) if lats else 0,
                "max_latency_ms": max(lats) if lats else 0,
                "error_breakdown": s["error_types"],
            })

        providers.sort(key=lambda x: -x["health_score"])

        return {
            "window_hours": window_hours,
            "total_records": len(records),
            "providers": providers,
            "timestamp_utc": datetime.now(UTC).isoformat(),
        }

    def smart_order(self, base_order: list[str]) -> list[str]:
        """Reorder providers by health score, falling back to base_order for unknowns.

        Only reorders if enough data is available (_MIN_RECORDS_FOR_REORDER).
        Providers not in health data keep their base_order position.
        """
        scores = self.get_scores()

        # Check if we have enough data to reorder
        total_records = sum(
            1 for _ in self._load_records() if True
        )
        if total_records < _MIN_RECORDS_FOR_REORDER:
            return list(base_order)

        # Split into scored and unscored
        scored = [(name, scores[name]) for name in base_order if name in scores]
        unscored = [name for name in base_order if name not in scores]

        # Sort scored by health score descending
        scored.sort(key=lambda x: -x[1])

        # Interleave: scored first, then unscored in original order
        return [name for name, _ in scored] + unscored

    def compact(self, keep_hours: int = _DEFAULT_WINDOW_HOURS) -> int:
        """Remove records older than keep_hours. Returns number removed."""
        records = self._load_records(keep_hours)
        if not self.path.exists():
            return 0

        try:
            all_lines = self.path.read_text(encoding="utf-8").strip().split("\n")
            old_count = len([l for l in all_lines if l.strip()])
        except OSError:
            return 0

        new_count = len(records)
        removed = old_count - new_count

        if removed > 0:
            try:
                with self.path.open("w", encoding="utf-8") as f:
                    for rec in records:
                        f.write(json.dumps(rec, ensure_ascii=False) + "\n")
            except OSError:
                logger.warning("Failed to compact health records", exc_info=True)
                return 0

        return removed


path: src/free_search/providers.py
content: """Search provider implementations with a unified output contract."""

from __future__ import annotations

import logging
import time
import warnings
from dataclasses import dataclass
from typing import Any
from urllib.parse import parse_qs, unquote, urlparse
from xml.etree import ElementTree as ET

import requests
from bs4 import BeautifulSoup

logger = logging.getLogger(__name__)

DEFAULT_TIMEOUT_SECONDS = 12
DEFAULT_USER_AGENT = (
    "Mozilla/5.0 (X11; Linux x86_64) "
    "AppleWebKit/537.36 (KHTML, like Gecko) "
    "Chrome/122.0 Safari/537.36"
)
MAX_RESPONSE_BYTES = 5 * 1024 * 1024  # 5 MB guard against runaway responses


class ProviderError(Exception):
    """Base class for provider failures."""


class AuthError(ProviderError):
    """Raised when provider credentials are missing or rejected."""


class NetworkError(ProviderError):
    """Raised when upstream networking fails."""


class RateLimitError(ProviderError):
    """Raised when provider reports rate limiting."""


class QuotaExceededError(ProviderError):
    """Raised when provider quota has been exhausted."""


class ParseError(ProviderError):
    """Raised when an upstream response cannot be parsed safely."""


class UpstreamError(ProviderError):
    """Raised for non-recoverable upstream API errors."""


@dataclass(slots=True)
class SearchItem:
    """Unified search result item."""

    title: str
    url: str
    snippet: str
    source: str
    rank: int


class BaseProvider:
    """Base provider contract for search services."""

    name = "base"

    def __init__(
        self,
        *,
        config: dict[str, Any],
        session: requests.Session | None = None,
    ) -> None:
        self.config = config
        self.session = session or requests.Session()
        self.enabled = bool(config.get("enabled", True))
        self.timeout = int(config.get("timeout_seconds", DEFAULT_TIMEOUT_SECONDS))
        self.min_interval_seconds = float(config.get("min_interval_seconds", 0.0))
        self._last_request_ts = 0.0

    def is_enabled(self) -> bool:
        return self.enabled

    def maybe_sleep_for_rate_limit(self) -> float:
        if self.min_interval_seconds <= 0:
            return 0.0
        now = time.time()
        elapsed = now - self._last_request_ts
        remaining = self.min_interval_seconds - elapsed
        if remaining > 0:
            logger.debug("Provider %s sleeping %.2fs for pacing", self.name, remaining)
            time.sleep(remaining)
            return remaining
        return 0.0

    def _mark_request(self) -> None:
        self._last_request_ts = time.time()

    @staticmethod
    def _http_error_detail(response: requests.Response) -> str:
        try:
            payload = response.json()
            if isinstance(payload, dict):
                for key in ("error", "message", "detail"):
                    value = payload.get(key)
                    if value:
                        return str(value)
        except ValueError:
            pass
        text = (response.text or "").strip()
        if text:
            return text[:180]
        return ""

    @staticmethod
    def _guard_response_size(response: requests.Response) -> None:
        """Raise ParseError if the response body exceeds MAX_RESPONSE_BYTES."""
        cl = response.headers.get("content-length")
        if cl:
            try:
                if int(cl) > MAX_RESPONSE_BYTES:
                    raise ParseError(
                        f"Response content-length {cl} bytes exceeds {MAX_RESPONSE_BYTES} limit"
                    )
            except ValueError:
                pass
        if len(response.content) > MAX_RESPONSE_BYTES:
            raise ParseError(
                f"Response body {len(response.content)} bytes exceeds {MAX_RESPONSE_BYTES} limit"
            )

    def search(self, query: str, *, max_results: int) -> list[SearchItem]:
        raise NotImplementedError


class BraveProvider(BaseProvider):
    name = "brave"
    endpoint = "https://api.search.brave.com/res/v1/web/search"

    @staticmethod
    def _api_key_candidates(api_key: str) -> list[str]:
        key = api_key.strip()
        if not key:
            return []
        candidates = [key]
        if key.startswith("BSA"):
            candidates.append(f"BSa{key[3:]}")
        elif key.startswith("BSa"):
            candidates.append(f"BSA{key[3:]}")
        deduped: list[str] = []
        for candidate in candidates:
            if candidate not in deduped:
                deduped.append(candidate)
        return deduped

    def search(self, query: str, *, max_results: int) -> list[SearchItem]:
        [REDACTED_SECRET]"api_key") or "").strip()
        if not api_key:
            raise AuthError("Brave API key missing")

        self.maybe_sleep_for_rate_limit()
        params = {"q": query, "count": max_results}
        key_candidates = self._api_key_candidates(api_key)
        if not key_candidates:
            raise AuthError("Brave API key missing")

        response: requests.Response | None = None
        for index, candidate in enumerate(key_candidates):
            headers = {
                "Accept": "application/json",
                "X-Subscription-Token": candidate,
            }
            logger.debug(
                "Brave request: query=%r max_results=%s attempt=%s",
                query,
                max_results,
                index + 1,
            )
            try:
                response = self.session.get(
                    self.endpoint,
                    params=params,
                    headers=headers,
                    timeout=self.timeout,
                )
            except requests.RequestException as exc:
                raise NetworkError(f"Brave request failed: {exc}") from exc
            finally:
                self._mark_request()

            if response.status_code not in (401, 403):
                break
            if index + 1 < len(key_candidates):
                logger.warning(
                    "Brave auth failed with current key format; retrying with alternate prefix"
                )

        if response is None:
            raise UpstreamError("Brave request did not receive a response")

        if response.status_code in (401, 403):
            detail = self._http_error_detail(response)
            raise AuthError(
                f"Brave auth failed: HTTP {response.status_code}"
                + (f" ({detail})" if detail else "")
            )
        if response.status_code == 429:
            raise RateLimitError("Brave rate limited")
        if response.status_code >= 500:
            raise UpstreamError(f"Brave server error: HTTP {response.status_code}")
        if response.status_code >= 400:
            detail = self._http_error_detail(response)
            raise UpstreamError(
                f"Brave request rejected: HTTP {response.status_code}"
                + (f" ({detail})" if detail else "")
            )

        self._guard_response_size(response)
        try:
            payload = response.json()
        except ValueError as exc:
            raise ParseError("Brave returned non-JSON response") from exc

        web = payload.get("web", {})
        raw_results = web.get("results", [])
        items: list[SearchItem] = []
        for idx, row in enumerate(raw_results[:max_results], start=1):
            title = (row.get("title") or "").strip()
            url = (row.get("url") or "").strip()
            snippet = (row.get("description") or "").strip()
            if not (title and url):
                continue
            items.append(
                SearchItem(title=title, url=url, snippet=snippet, source=self.name, rank=idx)
            )
        logger.info("Brave returned %s results for query=%r", len(items), query)
        return items


class TavilyProvider(BaseProvider):
    name = "tavily"
    endpoint = "https://api.tavily.com/search"

    def search(self, query: str, *, max_results: int) -> list[SearchItem]:
        [REDACTED_SECRET]"api_key") or "").strip()
        if not api_key:
            raise AuthError("Tavily API key missing")

        self.maybe_sleep_for_rate_limit()
        body = {
            "api_key": api_key,
            "query": query,
            "max_results": max_results,
            "include_answer": False,
            "search_depth": self.config.get("search_depth", "basic"),
        }

        try:
            logger.debug("Tavily request: query=%r max_results=%s", query, max_results)
            response = self.session.post(self.endpoint, json=body, timeout=self.timeout)
        except requests.RequestException as exc:
            raise NetworkError(f"Tavily request failed: {exc}") from exc
        finally:
            self._mark_request()

        if response.status_code in (401, 403):
            detail = self._http_error_detail(response)
            raise AuthError(
                f"Tavily auth failed: HTTP {response.status_code}"
                + (f" ({detail})" if detail else "")
            )
        if response.status_code == 429:
            raise RateLimitError("Tavily rate limited")
        if response.status_code >= 500:
            raise UpstreamError(f"Tavily server error: HTTP {response.status_code}")
        if response.status_code >= 400:
            detail = self._http_error_detail(response)
            raise UpstreamError(
                f"Tavily request rejected: HTTP {response.status_code}"
                + (f" ({detail})" if detail else "")
            )

        self._guard_response_size(response)
        try:
            payload = response.json()
        except ValueError as exc:
            raise ParseError("Tavily returned non-JSON response") from exc

        raw_results = payload.get("results", [])
        items: list[SearchItem] = []
        for idx, row in enumerate(raw_results[:max_results], start=1):
            title = (row.get("title") or "").strip()
            url = (row.get("url") or "").strip()
            snippet = (row.get("content") or "").strip()
            if not (title and url):
                continue
            items.append(
                SearchItem(title=title, url=url, snippet=snippet, source=self.name, rank=idx)
            )
        logger.info("Tavily returned %s results for query=%r", len(items), query)
        return items


class DuckDuckGoProvider(BaseProvider):
    name = "duckduckgo"
    endpoint = "https://duckduckgo.com/html/"

    @staticmethod
    def _extract_target_url(href: str) -> str:
        raw_href = href.strip()
        if not raw_href:
            return ""
        if raw_href.startswith("//"):
            raw_href = f"https:{raw_href}"

        parsed = urlparse(raw_href)
        if "duckduckgo.com" in parsed.netloc and parsed.path.startswith("/l/"):
            target = parse_qs(parsed.query).get("uddg", [""])[0]
            if target:
                return unquote(target).strip()
        return raw_href

    def search(self, query: str, *, max_results: int) -> list[SearchItem]:
        self.maybe_sleep_for_rate_limit()
        params = {"q": query}
        headers = {"User-Agent": DEFAULT_USER_AGENT}

        try:
            logger.debug("DuckDuckGo HTML request: query=%r max_results=%s", query, max_results)
            response = self.session.get(
                self.endpoint, params=params, headers=headers, timeout=self.timeout
            )
        except requests.RequestException as exc:
            raise NetworkError(f"DuckDuckGo request failed: {exc}") from exc
        finally:
            self._mark_request()

        if response.status_code == 429:
            raise RateLimitError("DuckDuckGo rate limited")
        if response.status_code >= 500:
            raise UpstreamError(f"DuckDuckGo server error: HTTP {response.status_code}")
        if response.status_code >= 400:
            detail = self._http_error_detail(response)
            raise UpstreamError(
                f"DuckDuckGo request rejected: HTTP {response.status_code}"
                + (f" ({detail})" if detail else "")
            )

        self._guard_response_size(response)
        soup = BeautifulSoup(response.text, "html.parser")
        node_selectors = (
            "div.result",
            "article[data-testid='result']",
            "li[data-layout='organic']",
            "div.results_links",
        )
        nodes: list[Any] = []
        for selector in node_selectors:
            nodes.extend(soup.select(selector))

        items: list[SearchItem] = []
        seen_urls: set[str] = set()
        rank = 0
        for node in nodes:
            link = (
                node.select_one("a.result__a")
                or node.select_one("a[data-testid='result-title-a']")
                or node.select_one("h2 a[href]")
            )
            if not link:
                continue
            href = self._extract_target_url(link.get("href") or "")
            title = link.get_text(" ", strip=True)
            snippet_node = (
                node.select_one("a.result__snippet")
                or node.select_one(".result__snippet")
                or node.select_one("[data-result='snippet']")
                or node.select_one("[data-testid='result-snippet']")
            )
            snippet = snippet_node.get_text(" ", strip=True) if snippet_node else ""
            if not (title and href) or href in seen_urls:
                continue
            seen_urls.add(href)
            rank += 1
            items.append(
                SearchItem(title=title, url=href, snippet=snippet, source=self.name, rank=rank)
            )
            if rank >= max_results:
                break

        # Return empty list (not an error) — router will continue to next provider
        if not items:
            logger.info("DuckDuckGo HTML returned no results for query=%r", query)
        else:
            logger.info("DuckDuckGo HTML returned %s results for query=%r", len(items), query)
        return items


class SerperProvider(BaseProvider):
    name = "serper"
    endpoint = "https://google.serper.dev/search"

    def search(self, query: str, *, max_results: int) -> list[SearchItem]:
        [REDACTED_SECRET]"api_key") or "").strip()
        if not api_key:
            raise AuthError("Serper API key missing")

        self.maybe_sleep_for_rate_limit()
        headers = {"X-API-KEY": api_key, "Content-Type": "application/json"}
        body = {"q": query, "num": max_results}

        try:
            logger.debug("Serper request: query=%r max_results=%s", query, max_results)
            response = self.session.post(
                self.endpoint, headers=headers, json=body, timeout=self.timeout
            )
        except requests.RequestException as exc:
            raise NetworkError(f"Serper request failed: {exc}") from exc
        finally:
            self._mark_request()

        if response.status_code in (401, 403):
            detail = self._http_error_detail(response)
            raise AuthError(
                f"Serper auth failed: HTTP {response.status_code}"
                + (f" ({detail})" if detail else "")
            )
        if response.status_code == 429:
            raise RateLimitError("Serper rate limited")
        if response.status_code >= 500:
            raise UpstreamError(f"Serper server error: HTTP {response.status_code}")
        if response.status_code >= 400:
            detail = self._http_error_detail(response)
            raise UpstreamError(
                f"Serper request rejected: HTTP {response.status_code}"
                + (f" ({detail})" if detail else "")
            )

        self._guard_response_size(response)
        try:
            payload = response.json()
        except ValueError as exc:
            raise ParseError("Serper returned non-JSON response") from exc

        raw_results = payload.get("organic", [])
        items: list[SearchItem] = []
        for idx, row in enumerate(raw_results[:max_results], start=1):
            title = (row.get("title") or "").strip()
            url = (row.get("link") or "").strip()
            snippet = (row.get("snippet") or "").strip()
            if not (title and url):
                continue
            items.append(
                SearchItem(title=title, url=url, snippet=snippet, source=self.name, rank=idx)
            )
        logger.info("Serper returned %s results for query=%r", len(items), query)
        return items


class SearchApiProvider(BaseProvider):
    name = "searchapi"
    endpoint = "https://www.searchapi.io/api/v1/search"

    def search(self, query: str, *, max_results: int) -> list[SearchItem]:
        [REDACTED_SECRET]"api_key") or "").strip()
        if not api_key:
            raise AuthError("SearchApi API key missing")

        self.maybe_sleep_for_rate_limit()
        params = {
            "engine": self.config.get("engine", "google"),
            "q": query,
            "num": max_results,
        }
        # Use Bearer token header only — avoids leaking key in query string / server logs
        headers = {"Authorization": f"Bearer {api_key}"}

        try:
            logger.debug("SearchApi request: query=%r max_results=%s", query, max_results)
            response = self.session.get(
                self.endpoint, params=params, headers=headers, timeout=self.timeout
            )
        except requests.RequestException as exc:
            raise NetworkError(f"SearchApi request failed: {exc}") from exc
        finally:
            self._mark_request()

        if response.status_code in (401, 403):
            detail = self._http_error_detail(response)
            raise AuthError(
                f"SearchApi auth failed: HTTP {response.status_code}"
                + (f" ({detail})" if detail else "")
            )
        if response.status_code == 429:
            raise RateLimitError("SearchApi rate limited")
        if response.status_code >= 500:
            raise UpstreamError(f"SearchApi server error: HTTP {response.status_code}")
        if response.status_code >= 400:
            detail = self._http_error_detail(response)
            raise UpstreamError(
                f"SearchApi request rejected: HTTP {response.status_code}"
                + (f" ({detail})" if detail else "")
            )

        self._guard_response_size(response)
        try:
            payload = response.json()
        except ValueError as exc:
            raise ParseError("SearchApi returned non-JSON response") from exc

        raw_results = payload.get("organic_results", []) or payload.get("results", [])
        items: list[SearchItem] = []
        for idx, row in enumerate(raw_results[:max_results], start=1):
            title = (row.get("title") or "").strip()
            url = (row.get("link") or row.get("url") or "").strip()
            snippet = (row.get("snippet") or row.get("description") or "").strip()
            if not (title and url):
                continue
            items.append(
                SearchItem(title=title, url=url, snippet=snippet, source=self.name, rank=idx)
            )
        logger.info("SearchApi returned %s results for query=%r", len(items), query)
        return items


class DuckDuckGoInstantProvider(BaseProvider):
    name = "duckduckgo_instant"
    endpoint = "https://api.duckduckgo.com/"

    def search(self, query: str, *, max_results: int) -> list[SearchItem]:
        self.maybe_sleep_for_rate_limit()
        params = {"q": query, "format": "json", "no_redirect": 1, "no_html": 1}
        headers = {"User-Agent": DEFAULT_USER_AGENT}

        try:
            logger.debug(
                "DuckDuckGo Instant request: query=%r max_results=%s", query, max_results
            )
            response = self.session.get(
                self.endpoint, params=params, headers=headers, timeout=self.timeout
            )
        except requests.RequestException as exc:
            raise NetworkError(f"DuckDuckGo Instant request failed: {exc}") from exc
        finally:
            self._mark_request()

        if response.status_code == 429:
            raise RateLimitError("DuckDuckGo Instant rate limited")
        if response.status_code >= 500:
            raise UpstreamError(f"DuckDuckGo Instant server error: HTTP {response.status_code}")
        if response.status_code >= 400:
            detail = self._http_error_detail(response)
            raise UpstreamError(
                f"DuckDuckGo Instant request rejected: HTTP {response.status_code}"
                + (f" ({detail})" if detail else "")
            )

        self._guard_response_size(response)
        try:
            payload = response.json()
        except ValueError as exc:
            raise ParseError("DuckDuckGo Instant returned non-JSON response") from exc

        flattened: list[tuple[str, str]] = []
        abstract_text = (payload.get("AbstractText") or "").strip()
        abstract_url = (payload.get("AbstractURL") or "").strip()
        if abstract_text and abstract_url:
            flattened.append((abstract_text, abstract_url))

        def _collect(rows: list[dict[str, Any]]) -> None:
            for row in rows:
                if "Topics" in row and isinstance(row.get("Topics"), list):
                    _collect(row["Topics"])
                    continue
                text = (row.get("Text") or "").strip()
                url = (row.get("FirstURL") or "").strip()
                if text and url:
                    flattened.append((text, url))

        _collect(payload.get("Results", []))
        _collect(payload.get("RelatedTopics", []))

        items: list[SearchItem] = []
        seen_urls: set[str] = set()
        for text, url in flattened:
            if url in seen_urls:
                continue
            seen_urls.add(url)
            title = text.split(" - ", 1)[0].strip() or text[:80]
            items.append(
                SearchItem(
                    title=title, url=url, snippet=text, source=self.name, rank=len(items) + 1
                )
            )
            if len(items) >= max_results:
                break

        logger.info("DuckDuckGo Instant returned %s results for query=%r", len(items), query)
        return items


class BingRSSProvider(BaseProvider):
    """Bing web search via RSS feed — no API key required, uses Bing's XML endpoint.

    This avoids JavaScript/bot-detection issues of HTML scraping by consuming
    Bing's built-in RSS output (format=rss), which returns clean XML with direct URLs.
    """

    name = "bing_html"   # Kept as "bing_html" for backwards-compatibility in configs
    endpoint = "https://www.bing.com/search"

    def search(self, query: str, *, max_results: int) -> list[SearchItem]:
        self.maybe_sleep_for_rate_limit()
        headers = {
            "User-Agent": DEFAULT_USER_AGENT,
            "Accept": "application/rss+xml,application/xml,text/xml,*/*;q=0.8",
            "Accept-Language": "en-US,en;q=0.9",
        }
        params = {"q": query, "format": "rss", "count": max_results}

        try:
            logger.debug("Bing RSS request: query=%r max_results=%s", query, max_results)
            response = self.session.get(
                self.endpoint, params=params, headers=headers, timeout=self.timeout
            )
        except requests.RequestException as exc:
            raise NetworkError(f"Bing RSS request failed: {exc}") from exc
        finally:
            self._mark_request()

        if response.status_code == 429:
            raise RateLimitError("Bing rate limited")
        if response.status_code >= 500:
            raise UpstreamError(f"Bing server error: HTTP {response.status_code}")
        if response.status_code >= 400:
            raise UpstreamError(f"Bing request rejected: HTTP {response.status_code}")

        self._guard_response_size(response)
        try:
            root = ET.fromstring(response.text)
        except ET.ParseError as exc:
            raise ParseError(f"Bing RSS returned unparseable XML: {exc}") from exc

        channel = root.find("channel")
        raw_items = channel.findall("item") if channel is not None else root.findall(".//item")

        items: list[SearchItem] = []
        seen_urls: set[str] = set()
        for idx, item in enumerate(raw_items[:max_results], start=1):
            title = (item.findtext("title") or "").strip()
            url = (item.findtext("link") or "").strip()
            snippet = (item.findtext("description") or "").strip()
            # Strip any HTML tags from snippet
            if "<" in snippet:
                snippet = BeautifulSoup(snippet, "html.parser").get_text(" ", strip=True)
            if not (title and url) or url in seen_urls:
                continue
            seen_urls.add(url)
            items.append(
                SearchItem(title=title, url=url, snippet=snippet, source=self.name, rank=idx)
            )

        if not items:
            logger.info("Bing RSS returned no results for query=%r", query)
        else:
            logger.info("Bing RSS returned %s results for query=%r", len(items), query)
        return items


class MojeekProvider(BaseProvider):
    """Mojeek web search — no API key required, independent index (not Google/Bing).

    Mojeek is a privacy-focused search engine with its own web crawler and index.
    Provides a different result set from Google/Bing-based providers.
    HTML scraping of public search results (no API key needed for basic use).
    Optional: use Mojeek's JSON API with MOJEEK_API_KEY for higher limits.
    """

    name = "mojeek"
    endpoint = "https://www.mojeek.com/search"
    api_endpoint = "https://www.mojeek.com/api/v1/search"

    def search(self, query: str, *, max_results: int) -> list[SearchItem]:
        [REDACTED_SECRET]"api_key") or "").strip()

        self.maybe_sleep_for_rate_limit()

        if [REDACTED_SECRET] self._search_api(query, max_results=max_results, [REDACTED_SECRET])
        return self._search_html(query, max_results=max_results)

    def _search_api(self, query: str, *, max_results: int, api_key: str) -> list[SearchItem]:
        """Use official Mojeek JSON API (requires MOJEEK_API_KEY)."""
        params = {"q": query, "api_key": api_key, "fmt": "json", "nb": max_results}
        try:
            logger.debug("Mojeek API request: query=%r max_results=%s", query, max_results)
            response = self.session.get(self.api_endpoint, params=params, timeout=self.timeout)
        except requests.RequestException as exc:
            raise NetworkError(f"Mojeek API request failed: {exc}") from exc
        finally:
            self._mark_request()

        if response.status_code in (401, 403):
            raise AuthError(f"Mojeek API auth failed: HTTP {response.status_code}")
        if response.status_code == 429:
            raise RateLimitError("Mojeek rate limited")
        if response.status_code >= 500:
            raise UpstreamError(f"Mojeek server error: HTTP {response.status_code}")
        if response.status_code >= 400:
            raise UpstreamError(f"Mojeek API rejected: HTTP {response.status_code}")

        self._guard_response_size(response)
        try:
            payload = response.json()
        except ValueError as exc:
            raise ParseError("Mojeek API returned non-JSON") from exc

        raw_results = payload.get("results", payload if isinstance(payload, list) else [])
        items: list[SearchItem] = []
        for idx, row in enumerate(raw_results[:max_results], start=1):
            title = (row.get("title") or "").strip()
            url = (row.get("link") or row.get("url") or "").strip()
            snippet = (row.get("desc") or row.get("snippet") or "").strip()
            if not (title and url):
                continue
            items.append(SearchItem(title=title, url=url, snippet=snippet, source=self.name, rank=idx))
        logger.info("Mojeek API returned %s results for query=%r", len(items), query)
        return items

    def _search_html(self, query: str, *, max_results: int) -> list[SearchItem]:
        """Fallback: scrape Mojeek HTML search results."""
        headers = {
            "User-Agent": DEFAULT_USER_AGENT,
            "Accept": "text/html,application/xhtml+xml;q=0.9,*/*;q=0.8",
            "Accept-Language": "en-US,en;q=0.9",
        }
        params = {"q": query, "nb": max_results}
        try:
            logger.debug("Mojeek HTML request: query=%r max_results=%s", query, max_results)
            response = self.session.get(self.endpoint, params=params, headers=headers, timeout=self.timeout)
        except requests.RequestException as exc:
            raise NetworkError(f"Mojeek HTML request failed: {exc}") from exc
        finally:
            self._mark_request()

        if response.status_code == 429:
            raise RateLimitError("Mojeek rate limited")
        if response.status_code >= 500:
            raise UpstreamError(f"Mojeek server error: HTTP {response.status_code}")
        if response.status_code >= 400:
            raise UpstreamError(f"Mojeek request rejected: HTTP {response.status_code}")

        self._guard_response_size(response)
        soup = BeautifulSoup(response.text, "html.parser")

        items: list[SearchItem] = []
        seen_urls: set[str] = set()
        rank = 0
        for li in soup.select("ul.results-standard li"):
            # Title and URL from h2 > a.title
            h2 = li.select_one("h2 a.title") or li.select_one("h2 a[href]")
            if not h2:
                continue
            title = h2.get_text(" ", strip=True)
            url = (h2.get("href") or "").strip()
            # Snippet from p.s (Mojeek's description class)
            snippet_node = li.select_one("p.s")
            snippet = snippet_node.get_text(" ", strip=True) if snippet_node else ""
            if not (title and url) or not url.startswith("http") or url in seen_urls:
                continue
            seen_urls.add(url)
            rank += 1
            items.append(
                SearchItem(title=title, url=url, snippet=snippet, source=self.name, rank=rank)
            )
            if rank >= max_results:
                break

        if not items:
            logger.info("Mojeek HTML returned no results for query=%r", query)
        else:
            logger.info("Mojeek HTML returned %s results for query=%r", len(items), query)
        return items


class SearXNGProvider(BaseProvider):
    """SearXNG meta-search (self-hosted or private instance only) — no API key required.

    Public instances are typically rate-limited for server-to-server requests.
    Configure your own self-hosted instance via the `endpoint` config key.
    See: https://searxng.github.io/searxng/
    """

    name = "searxng"

    def search(self, query: str, *, max_results: int) -> list[SearchItem]:
        base = (self.config.get("endpoint") or "").rstrip("/")
        if not base:
            raise AuthError("SearXNG endpoint not configured (set searxng.endpoint in providers.yaml)")

        url = f"{base}/search"
        categories = self.config.get("categories", "general")

        self.maybe_sleep_for_rate_limit()
        params = {"q": query, "format": "json", "categories": categories}
        headers = {"User-Agent": DEFAULT_USER_AGENT, "Accept": "application/json"}

        try:
            logger.debug("SearXNG request: endpoint=%s query=%r", base, query)
            response = self.session.get(url, params=params, headers=headers, timeout=self.timeout)
        except requests.RequestException as exc:
            raise NetworkError(f"SearXNG request failed: {exc}") from exc
        finally:
            self._mark_request()

        if response.status_code == 429:
            raise RateLimitError("SearXNG rate limited")
        if response.status_code in (403, 401):
            raise UpstreamError(f"SearXNG returned {response.status_code} — JSON API may be disabled on this instance")
        if response.status_code >= 500:
            raise UpstreamError(f"SearXNG server error: HTTP {response.status_code}")
        if response.status_code >= 400:
            raise UpstreamError(f"SearXNG request rejected: HTTP {response.status_code}")

        self._guard_response_size(response)
        try:
            payload = response.json()
        except ValueError as exc:
            raise ParseError("SearXNG returned non-JSON (enable format=json on your instance)") from exc

        raw_results = payload.get("results", [])
        items: list[SearchItem] = []
        seen_urls: set[str] = set()
        rank = 0
        for row in raw_results:
            title = (row.get("title") or "").strip()
            url_ = (row.get("url") or "").strip()
            snippet = (row.get("content") or "").strip()
            if not (title and url_) or url_ in seen_urls:
                continue
            seen_urls.add(url_)
            rank += 1
            items.append(
                SearchItem(title=title, url=url_, snippet=snippet, source=self.name, rank=rank)
            )
            if rank >= max_results:
                break

        if not items:
            logger.info("SearXNG returned no results for query=%r (endpoint=%s)", query, base)
        else:
            logger.info("SearXNG returned %s results for query=%r", len(items), query)
        return items


class WikipediaProvider(BaseProvider):
    """Wikipedia Search API — completely free, no API key, great for factual queries."""

    name = "wikipedia"

    def search(self, query: str, *, max_results: int) -> list[SearchItem]:
        lang = self.config.get("lang", "en")
        endpoint = f"https://{lang}.wikipedia.org/w/api.php"

        self.maybe_sleep_for_rate_limit()
        params = {
            "action": "query",
            "list": "search",
            "srsearch": query,
            "srlimit": max_results,
            "format": "json",
            "srprop": "snippet|titlesnippet",
            "utf8": 1,
        }
        headers = {"User-Agent": DEFAULT_USER_AGENT}

        try:
            logger.debug("Wikipedia request: query=%r max_results=%s lang=%s", query, max_results, lang)
            response = self.session.get(endpoint, params=params, headers=headers, timeout=self.timeout)
        except requests.RequestException as exc:
            raise NetworkError(f"Wikipedia request failed: {exc}") from exc
        finally:
            self._mark_request()

        if response.status_code == 429:
            raise RateLimitError("Wikipedia rate limited")
        if response.status_code >= 500:
            raise UpstreamError(f"Wikipedia server error: HTTP {response.status_code}")
        if response.status_code >= 400:
            raise UpstreamError(f"Wikipedia request rejected: HTTP {response.status_code}")

        self._guard_response_size(response)
        try:
            payload = response.json()
        except ValueError as exc:
            raise ParseError("Wikipedia returned non-JSON response") from exc

        search_results = payload.get("query", {}).get("search", [])
        items: list[SearchItem] = []
        for idx, row in enumerate(search_results[:max_results], start=1):
            title = (row.get("title") or "").strip()
            if not title:
                continue
            # Strip HTML tags from snippet
            raw_snippet = row.get("snippet") or ""
            snippet = BeautifulSoup(raw_snippet, "html.parser").get_text(" ", strip=True)
            page_url = f"https://{lang}.wikipedia.org/wiki/{title.replace(' ', '_')}"
            items.append(
                SearchItem(title=title, url=page_url, snippet=snippet, source=self.name, rank=idx)
            )

        if not items:
            logger.info("Wikipedia returned no results for query=%r", query)
        else:
            logger.info("Wikipedia returned %s results for query=%r", len(items), query)
        return items


class GoogleCSEProvider(BaseProvider):
    """Google Custom Search JSON API — 100 queries/day free tier. Requires GOOGLE_API_KEY + GOOGLE_CX."""

    name = "google_cse"
    endpoint = "https://www.googleapis.com/customsearch/v1"

    def search(self, query: str, *, max_results: int) -> list[SearchItem]:
        [REDACTED_SECRET]"api_key") or "").strip()
        cx = (self.config.get("cx") or "").strip()
        if not api_key:
            raise AuthError("Google CSE API key (GOOGLE_API_KEY) missing")
        if not cx:
            raise AuthError("Google CSE search engine ID (GOOGLE_CX) missing")

        # Google CSE returns at most 10 results per request
        num = min(max_results, 10)

        self.maybe_sleep_for_rate_limit()
        params = {"key": api_key, "cx": cx, "q": query, "num": num}

        try:
            logger.debug("Google CSE request: query=%r num=%s", query, num)
            response = self.session.get(self.endpoint, params=params, timeout=self.timeout)
        except requests.RequestException as exc:
            raise NetworkError(f"Google CSE request failed: {exc}") from exc
        finally:
            self._mark_request()

        if response.status_code in (401, 403):
            detail = self._http_error_detail(response)
            raise AuthError(
                f"Google CSE auth failed: HTTP {response.status_code}"
                + (f" ({detail})" if detail else "")
            )
        if response.status_code == 429:
            raise RateLimitError("Google CSE rate limited (daily quota likely exhausted)")
        if response.status_code >= 500:
            raise UpstreamError(f"Google CSE server error: HTTP {response.status_code}")
        if response.status_code >= 400:
            detail = self._http_error_detail(response)
            raise UpstreamError(
                f"Google CSE request rejected: HTTP {response.status_code}"
                + (f" ({detail})" if detail else "")
            )

        self._guard_response_size(response)
        try:
            payload = response.json()
        except ValueError as exc:
            raise ParseError("Google CSE returned non-JSON response") from exc

        raw_results = payload.get("items", [])
        items: list[SearchItem] = []
        for idx, row in enumerate(raw_results[:max_results], start=1):
            title = (row.get("title") or "").strip()
            url = (row.get("link") or "").strip()
            snippet = (row.get("snippet") or "").strip()
            if not (title and url):
                continue
            items.append(
                SearchItem(title=title, url=url, snippet=snippet, source=self.name, rank=idx)
            )
        logger.info("Google CSE returned %s results for query=%r", len(items), query)
        return items


class ExaProvider(BaseProvider):
    """Exa.ai neural/keyword search — 1000 searches/month free tier. Requires EXA_API_KEY."""

    name = "exa"
    endpoint = "https://api.exa.ai/search"

    def search(self, query: str, *, max_results: int) -> list[SearchItem]:
        [REDACTED_SECRET]"api_key") or "").strip()
        if not api_key:
            raise AuthError("Exa API key missing")

        self.maybe_sleep_for_rate_limit()
        headers = {
            "x-api-key": api_key,
            "Content-Type": "application/json",
            "Accept": "application/json",
        }
        body: dict[str, Any] = {
            "query": query,
            "numResults": max_results,
            "type": self.config.get("search_type", "auto"),
        }
        # Optionally include text snippets
        if self.config.get("include_text", True):
            body["contents"] = {"text": {"maxCharacters": 500}}

        try:
            logger.debug("Exa request: query=%r max_results=%s type=%s", query, max_results, body["type"])
            response = self.session.post(self.endpoint, json=body, headers=headers, timeout=self.timeout)
        except requests.RequestException as exc:
            raise NetworkError(f"Exa request failed: {exc}") from exc
        finally:
            self._mark_request()

        if response.status_code in (401, 403):
            detail = self._http_error_detail(response)
            raise AuthError(
                f"Exa auth failed: HTTP {response.status_code}"
                + (f" ({detail})" if detail else "")
            )
        if response.status_code == 429:
            raise RateLimitError("Exa rate limited")
        if response.status_code >= 500:
            raise UpstreamError(f"Exa server error: HTTP {response.status_code}")
        if response.status_code >= 400:
            detail = self._http_error_detail(response)
            raise UpstreamError(
                f"Exa request rejected: HTTP {response.status_code}"
                + (f" ({detail})" if detail else "")
            )

        self._guard_response_size(response)
        try:
            payload = response.json()
        except ValueError as exc:
            raise ParseError("Exa returned non-JSON response") from exc

        raw_results = payload.get("results", [])
        items: list[SearchItem] = []
        for idx, row in enumerate(raw_results[:max_results], start=1):
            title = (row.get("title") or "").strip()
            url = (row.get("url") or "").strip()
            # Exa returns content in a nested structure or at top level
            contents = row.get("contents") or {}
            snippet = (
                contents.get("text")
                or row.get("text")
                or row.get("excerpt")
                or row.get("summary")
                or ""
            ).strip()
            if len(snippet) > 500:
                snippet = snippet[:500].rsplit(" ", 1)[0] + "…"
            if not (title and url):
                continue
            items.append(
                SearchItem(title=title, url=url, snippet=snippet, source=self.name, rank=idx)
            )
        logger.info("Exa returned %s results for query=%r", len(items), query)
        return items


class BaiduProvider(BaseProvider):
    """Baidu Qianfan AI Search — good for Chinese-language content. Requires BAIDU_API_KEY."""

    name = "baidu"
    endpoint = "https://qianfan.baidubce.com/v2/ai_search/web_search"

    def search(self, query: str, *, max_results: int) -> list[SearchItem]:
        [REDACTED_SECRET]"api_key") or "").strip()
        if not api_key:
            raise AuthError("Baidu API key missing")

        self.maybe_sleep_for_rate_limit()
        headers = {
            "Authorization": f"Bearer {api_key}",
            "X-Appbuilder-From": "openclaw",
            "Content-Type": "application/json",
        }
        top_k = max(max_results, 5)
        body = {
            "messages": [{"content": query, "role": "user"}],
            "edition": self.config.get("edition", "standard"),
            "search_source": "baidu_search_v2",
            "resource_type_filter": [{"type": "web", "top_k": top_k}],
            "search_recency_filter": self.config.get("search_recency_filter", "year"),
            "safe_search": False,
        }

        try:
            logger.debug("Baidu request: query=%r max_results=%s", query, max_results)
            response = self.session.post(self.endpoint, json=body, headers=headers, timeout=self.timeout)
        except requests.RequestException as exc:
            raise NetworkError(f"Baidu request failed: {exc}") from exc
        finally:
            self._mark_request()

        if response.status_code in (401, 403):
            detail = self._http_error_detail(response)
            raise AuthError(
                f"Baidu auth failed: HTTP {response.status_code}"
                + (f" ({detail})" if detail else "")
            )
        if response.status_code == 429:
            raise RateLimitError("Baidu rate limited")
        if response.status_code >= 500:
            raise UpstreamError(f"Baidu server error: HTTP {response.status_code}")
        if response.status_code >= 400:
            detail = self._http_error_detail(response)
            raise UpstreamError(
                f"Baidu request rejected: HTTP {response.status_code}"
                + (f" ({detail})" if detail else "")
            )

        self._guard_response_size(response)
        try:
            payload = response.json()
        except ValueError as exc:
            raise ParseError("Baidu returned non-JSON response") from exc

        if isinstance(payload, dict) and "code" in payload:
            raise UpstreamError(f"Baidu API error: {payload.get('message', 'unknown error')}")

        raw_results = payload.get("references", [])
        items: list[SearchItem] = []
        for idx, row in enumerate(raw_results[:max_results], start=1):
            title = (row.get("title") or "").strip()
            url = (row.get("url") or "").strip()
            snippet = (
                row.get("content") or row.get("abstract") or row.get("site_name") or ""
            ).strip()
            if not (title and url):
                continue
            items.append(
                SearchItem(title=title, url=url, snippet=snippet, source=self.name, rank=idx)
            )
        logger.info("Baidu returned %s results for query=%r", len(items), query)
        return items


class YaCyProvider(BaseProvider):
    name = "yacy"
    endpoint = "http://localhost:8090/yacysearch.json"

    def search(self, query: str, *, max_results: int) -> list[SearchItem]:
        endpoint = self.config.get("endpoint") or self.endpoint

        self.maybe_sleep_for_rate_limit()
        params = {
            "query": query,
            "maximumRecords": max_results,
            "startRecord": 0,
            "resource": self.config.get("resource", "global"),
        }

        try:
            response = self.session.get(endpoint, params=params, timeout=self.timeout)
        except requests.RequestException as exc:
            raise NetworkError(f"YaCy request failed: {exc}") from exc
        finally:
            self._mark_request()

        if response.status_code in (401, 403):
            raise AuthError(f"YaCy auth failed: HTTP {response.status_code}")
        if response.status_code == 429:
            raise RateLimitError("YaCy rate limited")
        if response.status_code >= 500:
            raise UpstreamError(f"YaCy server error: HTTP {response.status_code}")
        if response.status_code >= 400:
            raise UpstreamError(f"YaCy request rejected: HTTP {response.status_code}")

        self._guard_response_size(response)
        try:
            payload = response.json()
        except ValueError as exc:
            raise ParseError("YaCy returned non-JSON response") from exc

        raw_items: list[dict[str, Any]] = []
        channels = payload.get("channels")
        if isinstance(channels, list):
            for channel in channels:
                if isinstance(channel, dict) and isinstance(channel.get("items"), list):
                    raw_items.extend(channel["items"])
        if not raw_items and isinstance(payload.get("items"), list):
            raw_items = payload["items"]

        items: list[SearchItem] = []
        for idx, row in enumerate(raw_items[:max_results], start=1):
            title = (row.get("title") or "").strip()
            url = (row.get("link") or row.get("url") or "").strip()
            snippet = (row.get("description") or row.get("snippet") or "").strip()
            if not (title and url):
                continue
            items.append(
                SearchItem(title=title, url=url, snippet=snippet, source=self.name, rank=idx)
            )
        return items


PROVIDER_REGISTRY = {
    BraveProvider.name: BraveProvider,
    TavilyProvider.name: TavilyProvider,
    DuckDuckGoProvider.name: DuckDuckGoProvider,
    DuckDuckGoInstantProvider.name: DuckDuckGoInstantProvider,
    SearchApiProvider.name: SearchApiProvider,
    YaCyProvider.name: YaCyProvider,
    SerperProvider.name: SerperProvider,
    BingRSSProvider.name: BingRSSProvider,
    MojeekProvider.name: MojeekProvider,
    SearXNGProvider.name: SearXNGProvider,
    WikipediaProvider.name: WikipediaProvider,
    GoogleCSEProvider.name: GoogleCSEProvider,
    ExaProvider.name: ExaProvider,
    BaiduProvider.name: BaiduProvider,
}


path: src/free_search/quality.py
content: """Search result quality optimization: relevance scoring, dedup, reranking."""

from __future__ import annotations

import re
from typing import Any
from urllib.parse import urlparse

# ── Tokenization ────────────────────────────────────────────────────────────

_TOKEN_PATTERN = re.compile(r"[\w\u4e00-\u9fff\u3040-\u30ff]+", re.UNICODE)


def _tokenize(text: str) -> set[str]:
    """Split text into lowercase token set (works for Latin + CJK)."""
    return {m.group().lower() for m in _TOKEN_PATTERN.finditer(text)} if text else set()


# ── Relevance Scoring ───────────────────────────────────────────────────────


def relevance_score(query: str, title: str, snippet: str) -> float:
    """Compute relevance of a result to the query (0.0–1.0).

    Weighted: title match 60%, snippet match 40%.
    """
    q_tokens = _tokenize(query)
    if not q_tokens:
        return 0.0

    t_tokens = _tokenize(title)
    s_tokens = _tokenize(snippet)

    title_overlap = len(q_tokens & t_tokens) / len(q_tokens) if q_tokens else 0.0
    snippet_overlap = len(q_tokens & s_tokens) / len(q_tokens) if q_tokens else 0.0

    return min(1.0, title_overlap * 0.6 + snippet_overlap * 0.4)


# ── Similarity ──────────────────────────────────────────────────────────────


def title_similarity(a: str, b: str) -> float:
    """Jaccard token similarity between two titles."""
    ta = _tokenize(a)
    tb = _tokenize(b)
    if not ta and not tb:
        return 1.0
    if not ta or not tb:
        return 0.0
    return len(ta & tb) / len(ta | tb)


# ── Domain Extraction ───────────────────────────────────────────────────────


def _extract_domain(url: str) -> str:
    try:
        netloc = urlparse(url).netloc.lower()
        # Strip www. prefix for grouping
        if netloc.startswith("www."):
            netloc = netloc[4:]
        return netloc
    except Exception:
        return ""


# ── Deduplication ────────────────────────────────────────────────────────────


def deduplicate_results(
    results: list[dict[str, Any]],
    *,
    title_threshold: float = 0.80,
) -> list[dict[str, Any]]:
    """Remove near-duplicate results by URL canonicalization + title similarity.

    Preserves order; keeps the first occurrence.
    """
    kept: list[dict[str, Any]] = []
    seen_urls: set[str] = set()
    kept_titles: list[str] = []

    for item in results:
        url = (item or {}).get("url", "")
        title = (item or {}).get("title", "")

        # URL-level dedup
        domain_path = _extract_domain(url) + urlparse(url).path.rstrip("/").lower()
        if domain_path in seen_urls:
            continue

        # Title similarity dedup
        is_dup = False
        for prev_title in kept_titles:
            if title_similarity(title, prev_title) >= title_threshold:
                is_dup = True
                break
        if is_dup:
            continue

        seen_urls.add(domain_path)
        kept_titles.append(title)
        kept.append(item)

    return kept


# ── Reranking ────────────────────────────────────────────────────────────────


def rerank_results(
    query: str,
    results: list[dict[str, Any]],
    *,
    max_per_domain: int = 3,
) -> list[dict[str, Any]]:
    """Rerank results by relevance + domain diversity.

    - Computes relevance_score for each result
    - Limits max_per_domain results from the same domain
    - Sorts by score descending, preserving ties by original order
    """
    scored: list[tuple[float, int, dict[str, Any]]] = []
    for idx, item in enumerate(results):
        title = (item or {}).get("title", "")
        snippet = (item or {}).get("snippet", "")
        score = relevance_score(query, title, snippet)
        # Bonus for longer snippets (information richness)
        snippet_len = len(snippet) if snippet else 0
        length_bonus = min(0.1, snippet_len / 1000.0)
        item["_relevance_score"] = round(score + length_bonus, 4)
        scored.append((score + length_bonus, idx, item))

    # Sort by score desc, then original position
    scored.sort(key=lambda x: (-x[0], x[1]))

    # Domain diversity filter
    domain_counts: dict[str, int] = {}
    output: list[dict[str, Any]] = []
    for _score, _idx, item in scored:
        domain = _extract_domain(item.get("url", ""))
        if domain:
            count = domain_counts.get(domain, 0)
            if count >= max_per_domain:
                continue
            domain_counts[domain] = count + 1
        output.append(item)

    return output


# ── Filtering ────────────────────────────────────────────────────────────────


def filter_low_quality(results: list[dict[str, Any]]) -> list[dict[str, Any]]:
    """Remove results that are clearly low quality."""
    filtered: list[dict[str, Any]] = []
    for item in results:
        title = (item or {}).get("title", "")
        snippet = (item or {}).get("snippet", "")
        url = (item or {}).get("url", "")

        # Skip results with no title or extremely short title
        if not title or len(title.strip()) < 3:
            continue
        # Skip results with no URL
        if not url:
            continue
        # Keep results even with empty snippet (some valid results lack snippets)
        # but deprioritize them in reranking via score

        filtered.append(item)
    return filtered


# ── Full Pipeline ────────────────────────────────────────────────────────────


def optimize_results(
    query: str,
    results: list[dict[str, Any]],
    *,
    max_per_domain: int = 3,
    title_dedup_threshold: float = 0.80,
) -> list[dict[str, Any]]:
    """Full quality optimization pipeline.

    1. Filter low-quality results
    2. Deduplicate (URL + title similarity)
    3. Rerank by relevance + domain diversity

    Returns optimized result list. Does not mutate input.
    """
    if not results:
        return []

    # Work on copies to avoid mutating input
    items = [dict(r) for r in results if r]

    items = filter_low_quality(items)
    items = deduplicate_results(items, title_threshold=title_dedup_threshold)
    items = rerank_results(query, items, max_per_domain=max_per_domain)

    # Clean up internal scoring keys before returning
    for item in items:
        item.pop("_relevance_score", None)

    return items


path: src/free_search/quota_real.py
content: """Fetch real provider quota/usage when supported by provider APIs."""

from __future__ import annotations

import json
import os
import time
from dataclasses import dataclass
from typing import Any

import requests

from .router import SearchRouter


@dataclass
class RealQuotaResult:
    provider: str
    supported: bool
    ok: bool
    remaining: int | None
    limit: int | None
    used: int | None
    unit: str | None
    detail: str | None
    raw: dict[str, Any] | None
    probe_consumes_request: bool = False


def _safe_int(value: Any) -> int | None:
    try:
        if value is None:
            return None
        return int(value)
    except (TypeError, ValueError):
        return None


def _get_searchapi_quota(api_key: str) -> RealQuotaResult:
    url = "https://www.searchapi.io/api/v1/me"
    headers = {"Authorization": f"Bearer {api_key}"}
    try:
        resp = requests.get(url, headers=headers, timeout=10)
    except requests.RequestException as exc:
        return RealQuotaResult(
            provider="searchapi",
            supported=True,
            ok=False,
            remaining=None,
            limit=None,
            used=None,
            unit="monthly_credits",
            detail=f"network_error: {exc}",
            raw=None,
        )

    if resp.status_code != 200:
        return RealQuotaResult(
            provider="searchapi",
            supported=True,
            ok=False,
            remaining=None,
            limit=None,
            used=None,
            unit="monthly_credits",
            detail=f"http_{resp.status_code}: {resp.text[:200]}",
            raw=None,
        )

    try:
        payload = resp.json()
    except ValueError:
        return RealQuotaResult(
            provider="searchapi",
            supported=True,
            ok=False,
            remaining=None,
            limit=None,
            used=None,
            unit="monthly_credits",
            detail="invalid_json",
            raw={"text": resp.text[:200]},
        )

    account = payload.get("account", {}) if isinstance(payload, dict) else {}
    remaining = _safe_int(account.get("remaining_credits"))
    limit = _safe_int(account.get("monthly_allowance"))
    used = _safe_int(account.get("current_month_usage"))
    return RealQuotaResult(
        provider="searchapi",
        supported=True,
        ok=True,
        remaining=remaining,
        limit=limit,
        used=used,
        unit="monthly_credits",
        detail=None,
        raw=payload if isinstance(payload, dict) else {"raw": payload},
    )


def _get_tavily_quota(api_key: str) -> RealQuotaResult:
    url = "https://api.tavily.com/usage"
    headers = {"Authorization": f"Bearer {api_key}"}
    params = {"api_key": api_key}

    # Try Authorization header first, then fallback to query param.
    try:
        resp = requests.get(url, headers=headers, timeout=10)
        if resp.status_code in (401, 403):
            resp = requests.get(url, params=params, timeout=10)
    except requests.RequestException as exc:
        return RealQuotaResult(
            provider="tavily",
            supported=True,
            ok=False,
            remaining=None,
            limit=None,
            used=None,
            unit="requests",
            detail=f"network_error: {exc}",
            raw=None,
        )

    if resp.status_code != 200:
        return RealQuotaResult(
            provider="tavily",
            supported=True,
            ok=False,
            remaining=None,
            limit=None,
            used=None,
            unit="requests",
            detail=f"http_{resp.status_code}: {resp.text[:200]}",
            raw=None,
        )

    try:
        payload = resp.json()
    except ValueError:
        return RealQuotaResult(
            provider="tavily",
            supported=True,
            ok=False,
            remaining=None,
            limit=None,
            used=None,
            unit="requests",
            detail="invalid_json",
            raw={"text": resp.text[:200]},
        )

    # Tavily response schema is not yet standardized in public docs.
    # We attempt to extract common fields but always return raw.
    remaining = _safe_int(payload.get("remaining")) if isinstance(payload, dict) else None
    limit = _safe_int(payload.get("limit")) if isinstance(payload, dict) else None
    used = _safe_int(payload.get("used")) if isinstance(payload, dict) else None

    return RealQuotaResult(
        provider="tavily",
        supported=True,
        ok=True,
        remaining=remaining,
        limit=limit,
        used=used,
        unit="requests",
        detail=None if (remaining is not None or limit is not None or used is not None) else "schema_unknown",
        raw=payload if isinstance(payload, dict) else {"raw": payload},
    )


def _get_brave_quota(api_key: str) -> RealQuotaResult:
    # Brave does not provide a dedicated quota endpoint; quota is available via headers.
    # We do a light probe (counts against quota) to read X-RateLimit-Remaining.
    url = "https://api.search.brave.com/res/v1/web/search"
    headers = {"Accept": "application/json", "X-Subscription-Token": api_key}
    params = {"q": "health", "count": 1}
    try:
        resp = requests.get(url, headers=headers, params=params, timeout=10)
    except requests.RequestException as exc:
        return RealQuotaResult(
            provider="brave",
            supported=True,
            ok=False,
            remaining=None,
            limit=None,
            used=None,
            unit="requests",
            detail=f"network_error: {exc}",
            raw=None,
            probe_consumes_request=True,
        )

    remaining = None
    limit = None
    try:
        remaining_header = resp.headers.get("X-RateLimit-Remaining")
        limit_header = resp.headers.get("X-RateLimit-Limit")
        if remaining_header:
            parts = [p.strip() for p in remaining_header.split(",")]
            remaining = _safe_int(parts[-1]) if parts else None
        if limit_header:
            parts = [p.strip() for p in limit_header.split(",")]
            limit = _safe_int(parts[-1]) if parts else None
    except Exception:
        pass

    ok = resp.status_code == 200
    detail = None if ok else f"http_{resp.status_code}: {resp.text[:200]}"
    return RealQuotaResult(
        provider="brave",
        supported=True,
        ok=ok,
        remaining=remaining,
        limit=limit,
        used=None,
        unit="requests",
        detail=detail,
        raw={"headers": dict(resp.headers)},
        probe_consumes_request=True,
    )


def _get_serper_quota(api_key: str) -> RealQuotaResult:
    # Serper does not expose a documented usage endpoint.
    return RealQuotaResult(
        provider="serper",
        supported=False,
        ok=False,
        remaining=None,
        limit=None,
        used=None,
        unit="requests",
        detail="no_public_quota_endpoint",
        raw=None,
    )


def _get_duckduckgo_quota() -> RealQuotaResult:
    return RealQuotaResult(
        provider="duckduckgo",
        supported=False,
        ok=False,
        remaining=None,
        limit=None,
        used=None,
        unit=None,
        detail="no_quota_applies",
        raw=None,
    )


def get_real_quota(
    *,
    config_path: str | None = None,
    probe_brave: bool = False,
) -> dict[str, Any]:
    router = SearchRouter(config_path=config_path)
    providers_cfg = router.config.get("providers", {})
    results: list[RealQuotaResult] = []

    for name in router.order:
        cfg = providers_cfg.get(name, {})
        enabled = bool(cfg.get("enabled", True))
        if not enabled:
            results.append(
                RealQuotaResult(
                    provider=name,
                    supported=False,
                    ok=False,
                    remaining=None,
                    limit=None,
                    used=None,
                    unit=None,
                    detail="disabled",
                    raw=None,
                )
            )
            continue

        [REDACTED_SECRET]"api_key") or "").strip()

        if name == "tavily":
            if not [REDACTED_SECRET]
                    RealQuotaResult(
                        provider=name,
                        supported=True,
                        ok=False,
                        remaining=None,
                        limit=None,
                        used=None,
                        unit="requests",
                        detail="missing_api_key",
                        raw=None,
                    )
                )
                continue
            results.append(_get_tavily_quota(api_key))
        elif name == "searchapi":
            if not [REDACTED_SECRET]
                    RealQuotaResult(
                        provider=name,
                        supported=True,
                        ok=False,
                        remaining=None,
                        limit=None,
                        used=None,
                        unit="monthly_credits",
                        detail="missing_api_key",
                        raw=None,
                    )
                )
                continue
            results.append(_get_searchapi_quota(api_key))
        elif name == "brave":
            if not [REDACTED_SECRET]
                    RealQuotaResult(
                        provider=name,
                        supported=True,
                        ok=False,
                        remaining=None,
                        limit=None,
                        used=None,
                        unit="requests",
                        detail="missing_api_key",
                        raw=None,
                    )
                )
                continue
            if probe_brave:
                results.append(_get_brave_quota(api_key))
            else:
                results.append(
                    RealQuotaResult(
                        provider=name,
                        supported=True,
                        ok=False,
                        remaining=None,
                        limit=None,
                        used=None,
                        unit="requests",
                        detail="requires_probe_for_headers",
                        raw=None,
                        probe_consumes_request=True,
                    )
                )
        elif name == "serper":
            if not [REDACTED_SECRET]
                    RealQuotaResult(
                        provider=name,
                        supported=True,
                        ok=False,
                        remaining=None,
                        limit=None,
                        used=None,
                        unit="requests",
                        detail="missing_api_key",
                        raw=None,
                    )
                )
                continue
            results.append(_get_serper_quota(api_key))
        elif name.startswith("duckduckgo"):
            results.append(_get_duckduckgo_quota())
        elif name in ("bing_html", "mojeek", "searxng", "wikipedia"):
            results.append(
                RealQuotaResult(
                    provider=name,
                    supported=False,
                    ok=False,
                    remaining=None,
                    limit=None,
                    used=None,
                    unit=None,
                    detail="no_quota_applies",
                    raw=None,
                )
            )
        elif name in ("google_cse", "exa", "baidu", "yacy"):
            results.append(
                RealQuotaResult(
                    provider=name,
                    supported=False,
                    ok=False,
                    remaining=None,
                    limit=None,
                    used=None,
                    unit=None,
                    detail="no_public_quota_endpoint",
                    raw=None,
                )
            )
        else:
            results.append(
                RealQuotaResult(
                    provider=name,
                    supported=False,
                    ok=False,
                    remaining=None,
                    limit=None,
                    used=None,
                    unit=None,
                    detail="unsupported_provider",
                    raw=None,
                )
            )

    return {
        "timestamp": int(time.time()),
        "providers": [result.__dict__ for result in results],
    }



path: src/free_search/router.py
content: """Provider routing, failover, and quota/rate awareness."""

from __future__ import annotations

import json
import logging
import os
import re
import time
from dataclasses import asdict
from datetime import UTC, datetime
from pathlib import Path
from typing import Any

import yaml

from .providers import (
    AuthError,
    NetworkError,
    ParseError,
    ProviderError,
    QuotaExceededError,
    RateLimitError,
    SearchItem,
    UpstreamError,
    PROVIDER_REGISTRY,
)
from .health import HealthTracker
from .quality import optimize_results

logger = logging.getLogger(__name__)
_UNRESOLVED_ENV_PATTERN = re.compile(r"^\$\{[A-Za-z_][A-Za-z0-9_]*\}$")

# Warn when a provider has consumed this fraction of its daily quota
QUOTA_WARN_THRESHOLD = 0.80


class SearchRouterError(Exception):
    """Raised when all providers are exhausted."""


class QuotaState:
    """Persistent provider usage tracker with day-level quotas."""

    def __init__(self, path: Path) -> None:
        self.path = path
        self.today = datetime.now(UTC).date().isoformat()
        self.state: dict[str, Any] = {"date": self.today, "providers": {}}
        self._load()

    def _load(self) -> None:
        if not self.path.exists():
            return
        try:
            data = json.loads(self.path.read_text(encoding="utf-8"))
        except (OSError, json.JSONDecodeError):
            logger.warning("Unable to read quota state at %s; resetting", self.path)
            return
        if data.get("date") != self.today:
            return
        self.state = data

    def save(self) -> None:
        try:
            self.path.parent.mkdir(parents=True, exist_ok=True)
            self.path.write_text(json.dumps(self.state, indent=2), encoding="utf-8")
        except OSError:
            logger.warning("Failed to persist quota state at %s", self.path, exc_info=True)

    def get_count(self, provider: str) -> int:
        pstate = self.state["providers"].get(provider, {})
        return int(pstate.get("requests", 0))

    def increment(self, provider: str) -> None:
        pstate = self.state["providers"].setdefault(provider, {})
        pstate["requests"] = int(pstate.get("requests", 0)) + 1
        pstate["last_request_ts"] = int(time.time())

    def reset(self, provider: str | None = None) -> None:
        if provider is None:
            self.state = {"date": self.today, "providers": {}}
            return
        self.state["providers"].pop(provider, None)


class SearchRouter:
    """Routes search requests through ordered providers with failover."""

    def __init__(self, config_path: str | os.PathLike[str] | None = None) -> None:
        default_config = Path(__file__).resolve().parents[2] / "config" / "providers.yaml"
        self.config_path = Path(config_path) if config_path else default_config
        self.config = self._load_config(self.config_path)
        self.order = self._load_order(self.config)

        quota_path = self.config.get("router", {}).get("quota_state_file", ".quota_state.json")
        if not os.path.isabs(quota_path):
            quota_path = str((self.config_path.parent / quota_path).resolve())
        self.quota = QuotaState(Path(quota_path))

        self.health = HealthTracker()
        self.order = self.health.smart_order(self.order)
        self.providers = self._build_providers()

    @staticmethod
    def _expand_env(value: Any) -> Any:
        if isinstance(value, str):
            expanded = os.path.expandvars(value)
            # Treat unresolved `${VAR}` placeholders as unset to avoid false auth attempts.
            if _UNRESOLVED_ENV_PATTERN.match(expanded):
                return ""
            return expanded
        if isinstance(value, dict):
            return {k: SearchRouter._expand_env(v) for k, v in value.items()}
        if isinstance(value, list):
            return [SearchRouter._expand_env(v) for v in value]
        return value

    def _load_config(self, path: Path) -> dict[str, Any]:
        if not path.exists():
            raise FileNotFoundError(f"Provider config not found: {path}")
        raw = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
        return self._expand_env(raw)

    def _load_order(self, config: dict[str, Any]) -> list[str]:
        order = config.get("router", {}).get("provider_order")
        if order:
            return [str(p).strip().lower() for p in order if str(p).strip()]
        return ["brave", "tavily", "duckduckgo", "serper"]

    def _build_providers(self) -> dict[str, Any]:
        configured = self.config.get("providers", {})
        providers: dict[str, Any] = {}
        for name in self.order:
            pconf = configured.get(name, {})
            cls = PROVIDER_REGISTRY.get(name)
            if not cls:
                logger.warning("Unknown provider in order list: %s", name)
                continue
            providers[name] = cls(config=pconf)
        return providers

    def _has_quota(self, name: str, provider_cfg: dict[str, Any]) -> bool:
        daily_quota = provider_cfg.get("daily_quota")
        if daily_quota is None:
            return True
        daily_quota = int(daily_quota)
        used = self.quota.get_count(name)
        if daily_quota > 0 and used >= daily_quota * QUOTA_WARN_THRESHOLD:
            pct = round(used / daily_quota * 100, 1)
            logger.warning(
                "Provider %s quota at %.1f%% (%d/%d) — nearing daily limit",
                name, pct, used, daily_quota,
            )
        return used < daily_quota

    def _try_provider(
        self,
        name: str,
        provider: Any,
        query: str,
        max_results: int,
    ) -> tuple[list[SearchItem] | None, dict[str, Any], bool]:
        """Attempt one provider search.

        Returns:
            (items, attempt_record, count_quota)
            - items=None means a hard failure (skip to next provider)
            - items=[]   means no results (skip to next provider, quota consumed)
            - items=[..] means success
            - count_quota: True if the HTTP request reached the provider
        """
        started = time.perf_counter()

        try:
            items: list[SearchItem] = provider.search(query, max_results=max_results)
        except (AuthError, NetworkError) as exc:
            # AuthError  → API rejected the key; provider quota was NOT consumed
            # NetworkError → request never reached the provider
            elapsed_ms = int((time.perf_counter() - started) * 1000)
            logger.warning("Provider %s failed (%s): %s", name, type(exc).__name__, exc)
            return (
                None,
                {
                    "provider": name,
                    "status": "failed",
                    "reason": type(exc).__name__,
                    "detail": str(exc),
                    "latency_ms": elapsed_ms,
                },
                False,  # do NOT count quota
            )
        except ProviderError as exc:
            # RateLimitError / UpstreamError / ParseError / QuotaExceededError
            # Request reached the provider → count quota
            elapsed_ms = int((time.perf_counter() - started) * 1000)
            logger.warning("Provider %s failed (%s): %s", name, type(exc).__name__, exc)
            return (
                None,
                {
                    "provider": name,
                    "status": "failed",
                    "reason": type(exc).__name__,
                    "detail": str(exc),
                    "latency_ms": elapsed_ms,
                },
                True,  # count quota — request was made
            )
        except Exception as exc:  # pragma: no cover — defensive failover guard
            elapsed_ms = int((time.perf_counter() - started) * 1000)
            logger.exception("Provider %s crashed unexpectedly", name)
            return (
                None,
                {
                    "provider": name,
                    "status": "failed",
                    "reason": "unexpected_error",
                    "detail": str(exc),
                    "latency_ms": elapsed_ms,
                },
                False,
            )

        elapsed_ms = int((time.perf_counter() - started) * 1000)
        if not items:
            return (
                [],
                {
                    "provider": name,
                    "status": "failed",
                    "reason": "empty_results",
                    "detail": "provider returned zero results",
                    "latency_ms": elapsed_ms,
                },
                True,  # request was made
            )

        return (
            items,
            {
                "provider": name,
                "status": "ok",
                "result_count": len(items),
                "latency_ms": elapsed_ms,
            },
            True,
        )

    def search(self, query: str, *, max_results: int = 8) -> dict[str, Any]:
        if max_results <= 0:
            raise ValueError("max_results must be > 0")
        attempted: list[dict[str, Any]] = []
        providers_cfg = self.config.get("providers", {})

        for name in self.order:
            provider = self.providers.get(name)
            cfg = providers_cfg.get(name, {})

            if not provider:
                attempted.append({"provider": name, "status": "skipped", "reason": "not_configured"})
                continue
            if not provider.is_enabled():
                attempted.append({"provider": name, "status": "skipped", "reason": "disabled"})
                continue
            if not self._has_quota(name, cfg):
                attempted.append({"provider": name, "status": "failed", "reason": "quota_exceeded"})
                continue

            logger.info("Attempting provider=%s for query=%r", name, query)
            items, attempt, count_quota = self._try_provider(name, provider, query, max_results)

            if count_quota:
                self.quota.increment(name)
                self.quota.save()

            # Record health data
            self.health.record(
                name,
                success=items is not None and len(items) > 0,
                latency_ms=attempt.get("latency_ms", 0),
                error_type=attempt.get("reason") if attempt.get("status") == "failed" else None,
            )

            attempted.append(attempt)

            if not items:  # None (error) or [] (empty results) → try next
                continue

            result_dicts = [asdict(item) for item in items]
            optimized = optimize_results(query, result_dicts)

            return {
                "query": query,
                "provider": name,
                "results": optimized,
                "meta": {
                    "attempted": attempted,
                    "quota": self._quota_snapshot(),
                    "timestamp_utc": datetime.now(UTC).isoformat(),
                    "quality": {
                        "raw_count": len(result_dicts),
                        "optimized_count": len(optimized),
                    },
                },
            }

        logger.error("All providers exhausted for query=%r", query)
        raise SearchRouterError(
            "All providers failed. "
            + " | ".join(f"{x['provider']}:{x.get('reason', 'unknown')}" for x in attempted)
        )

    def _quota_snapshot(self) -> dict[str, Any]:
        providers_cfg = self.config.get("providers", {})
        snapshot: dict[str, Any] = {}
        for name in self.order:
            used = self.quota.get_count(name)
            daily_quota = providers_cfg.get(name, {}).get("daily_quota")
            snapshot[name] = {
                "used_today": used,
                "daily_quota": int(daily_quota) if daily_quota is not None else None,
                "remaining": None if daily_quota is None else max(int(daily_quota) - used, 0),
            }
        return snapshot

    @staticmethod
    def _percentage_used(used: int, daily_quota: int | None) -> float | None:
        if daily_quota is None or daily_quota <= 0:
            return None
        return round((used / daily_quota) * 100, 2)

    def quota_status(self) -> dict[str, Any]:
        providers_cfg = self.config.get("providers", {})
        providers: list[dict[str, Any]] = []
        for name in self.order:
            used = self.quota.get_count(name)
            daily_quota_raw = providers_cfg.get(name, {}).get("daily_quota")
            daily_quota = int(daily_quota_raw) if daily_quota_raw is not None else None
            remaining = None if daily_quota is None else max(daily_quota - used, 0)
            providers.append(
                {
                    "provider": name,
                    "used_today": used,
                    "remaining": remaining,
                    "daily_quota": daily_quota,
                    "percentage_used": self._percentage_used(used, daily_quota),
                }
            )
        return {
            "date": self.quota.state.get("date", self.quota.today),
            "providers": providers,
        }

    def reset_quota(self, provider: str | None = None) -> dict[str, Any]:
        self.quota.reset(provider)
        self.quota.save()
        return self.quota_status()


path: src/free_search/storage.py
content: """Persistence utilities for search cache/index/report lifecycle."""

from __future__ import annotations

import hashlib
import json
import os
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import Any


def _workspace_root() -> Path:
    # src/free_search/storage.py -> free-search-aggregator/
    return Path(__file__).resolve().parents[2]


def _memory_root() -> Path:
    default_root = (_workspace_root().parent / "memory").resolve()
    custom = os.getenv("FREE_SEARCH_MEMORY_DIR", "").strip()
    if not custom:
        return default_root

    candidate = Path(custom).expanduser().resolve()
    allow_any = os.getenv([REDACTED_SECRET], "").strip() == "1"

    # Safety guard: by default only allow paths under workspace/memory.
    # To override intentionally, set FREE_SEARCH_ALLOW_ANY_MEMORY_DIR=1.
    if not allow_any:
        try:
            candidate.relative_to(default_root)
        except ValueError as exc:
            raise ValueError(
                "Unsafe FREE_SEARCH_MEMORY_DIR. Must be under workspace memory/ "
                "or set FREE_SEARCH_ALLOW_ANY_MEMORY_DIR=1 to override."
            ) from exc

    return candidate


def _paths() -> dict[str, Path]:
    root = _memory_root()
    return {
        "root": root,
        "cache": root / "search-cache",
        "index": root / "search-index" / "search-index.jsonl",
        "reports": root / "search-reports",
    }


def _slug(s: str, max_len: int = 80) -> str:
    out = []
    for ch in (s or ""):
        if ch.isalnum() or ch in ("-", "_", " "):
            out.append(ch)
        else:
            out.append("_")
    text = "".join(out).strip().replace(" ", "-")
    while "--" in text:
        text = text.replace("--", "-")
    return (text[:max_len] or "query").strip("-_") or "query"


def _now() -> datetime:
    return datetime.now(UTC)


def _extract_results(payload: dict[str, Any]) -> list[dict[str, Any]]:
    if "merged_results" in payload and isinstance(payload.get("merged_results"), list):
        return payload.get("merged_results", [])
    if isinstance(payload.get("results"), list):
        return payload.get("results", [])
    return []


def _build_report_markdown(query: str, payload: dict[str, Any], *, mode: str, ts: datetime) -> str:
    results = _extract_results(payload)
    lines = [
        f"# Search Report - {query}",
        "",
        f"- mode: {mode}",
        f"- timestamp_utc: {ts.isoformat()}",
        f"- result_count: {len(results)}",
        "",
        "## Top Results",
        "",
    ]
    for i, item in enumerate(results[:20], start=1):
        title = (item or {}).get("title", "(no title)")
        url = (item or {}).get("url", "")
        snippet = (item or {}).get("snippet", "")
        lines.append(f"{i}. [{title}]({url})")
        if snippet:
            lines.append(f"   - {snippet[:220]}")
    lines.append("")
    lines.append("---")
    lines.append("")
    lines.append("> Auto-generated by free-search-aggregator")
    return "\n".join(lines)


def persist_search_payload(*, query: str, payload: dict[str, Any], mode: str) -> dict[str, str]:
    ts = _now()
    date_str = ts.date().isoformat()
    p = _paths()

    p["cache"].mkdir(parents=True, exist_ok=True)
    p["reports"].mkdir(parents=True, exist_ok=True)
    p["index"].parent.mkdir(parents=True, exist_ok=True)

    digest = hashlib.sha256(f"{mode}|{query}|{ts.timestamp()}".encode("utf-8")).hexdigest()[:10]
    slug = _slug(query)

    cache_dir = p["cache"] / date_str
    cache_dir.mkdir(parents=True, exist_ok=True)
    cache_path = cache_dir / f"{slug}-{digest}.json"
    cache_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")

    report_dir = p["reports"] / date_str
    report_dir.mkdir(parents=True, exist_ok=True)
    report_path = report_dir / f"{slug}-{digest}.md"
    report_path.write_text(_build_report_markdown(query, payload, mode=mode, ts=ts), encoding="utf-8")

    results = _extract_results(payload)
    top_urls = []
    for item in results[:10]:
        url = (item or {}).get("url")
        if url:
            top_urls.append(url)

    index_line = {
        "ts": ts.isoformat(),
        "date": date_str,
        "mode": mode,
        "query": query,
        "hash": digest,
        "result_count": len(results),
        "cache_path": str(cache_path),
        "report_path": str(report_path),
        "top_urls": top_urls,
    }
    with p["index"].open("a", encoding="utf-8") as f:
        f.write(json.dumps(index_line, ensure_ascii=False) + "\n")

    return {
        "cache_path": str(cache_path),
        "report_path": str(report_path),
        "index_path": str(p["index"]),
    }


def run_gc(*, cache_days: int = 14, report_days: int | None = None) -> dict[str, Any]:
    now = _now()
    p = _paths()
    removed = {"cache_files": 0, "report_files": 0}

    def _remove_old_files(base: Path, days: int, key: str) -> None:
        if not base.exists() or days <= 0:
            return
        cutoff = now - timedelta(days=days)
        for file in base.rglob("*"):
            if not file.is_file():
                continue
            mtime = datetime.fromtimestamp(file.stat().st_mtime, tz=UTC)
            if mtime < cutoff:
                try:
                    file.unlink()
                    removed[key] += 1
                except OSError:
                    pass

    _remove_old_files(p["cache"], cache_days, "cache_files")
    if isinstance(report_days, int):
        _remove_old_files(p["reports"], report_days, "report_files")

    return {
        "ok": True,
        "memory_root": str(p["root"]),
        "removed": removed,
        "cache_days": cache_days,
        "report_days": report_days,
    }


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)