Skill: robbyczgw-cla/web-search-plus (version 3.1.0)

--- name: web-search-plus version: 3.1.0 description: Unified multi-provider web search and URL extraction skill with intelligent auto-routing across Serper, Brave, Tavily, Querit, Linkup, Exa, Firecrawl, Perplexity, You.com, SearXNG, and SerpBase. tags: [search, web-search, web-extract, serper, brave, tavily, querit, linkup, exa, firecrawl, perplexity, you, searxng, serpbase, google, multilingual-search, research, semantic-search, auto-routing, multi-provider, shopping, rag, free-tier, privacy, self-hosted, kilo] metadata: {"openclaw":{"requires":{"bins":["python3","bash"],"env":{"SERPER_API_KEY":"optional","BRAVE_API_KEY":"optional","TAVILY_API_KEY":"optional","QUERIT_API_KEY":"optional","LINKUP_API_KEY":"optional","EXA_API_KEY":"optional","FIRECRAWL_API_KEY":"optional","PERPLEXITY_API_KEY":"optional — direct Perplexity provider credential","KILOCODE_API_KEY":"optional — alternative Perplexity provider via Kilo Gateway","YOU_API_KEY":"optional","SEARXNG_INSTANCE_URL":"optional","SERPBASE_API_KEY":"optional — explicit/fallback-only Google SERP provider with prepaid credits"},"note":"Only ONE provider key or SEARXNG_INSTANCE_URL is needed for search. Extraction requires one of Firecrawl, Linkup, Tavily, Exa, or You.com."}}} --- # Web Search Plus **Stop choosing search providers. Let the skill do it for you.** This skill now connects you to **11 search providers** and adds a companion extraction flow for pulling content from URLs. Broad web query? → Brave or Serper. Research question? → Tavily or Exa. Need citations and grounding? → Linkup. Want scrape-ready content? → Firecrawl. Prefer privacy? → SearXNG. Need low-cost Google SERP with prepaid credits? → SerpBase (explicit/fallback-only). --- ## ✨ What Makes This Different? - **Just search** — no need to think about which provider to use - **Smart routing** — query analysis picks the best provider automatically - **11 providers, 1 interface** — general web, research, semantic discovery, direct answers, privacy-first, prepaid-credits, and extraction-capable providers together - **URL extraction included** — pull markdown/HTML content with fallback across five providers - **Works with just 1 credential** — start with any single provider, add more later - **Free/self-hosted options available** — SearXNG can run at $0 API cost --- ## 🚀 Quick Start ```bash # Interactive setup (recommended for first run) python3 scripts/setup.py # Or manually cp .env.example .env python3 scripts/search.py -q "latest OpenClaw release" python3 scripts/extract.py --url https://example.com ``` The wizard explains providers, collects keys, and sets defaults. --- ## 🔑 Providers ### Search providers - **Serper** — shopping, prices, local, and general Google-style results; fast broad fallback - **Brave** — independent web index and generic current-web queries; strong non-Google complement - **Tavily** — research, explanations, and synthesis; strong research routing - **Querit** — multilingual and international updates; good for cross-language recency - **Linkup** — source-grounded/citation-heavy search; evidence-first queries - **Exa** — semantic discovery, similar sites, and deep research; supports `deep` + `deep-reasoning` - **Firecrawl** — search with scrape-ready metadata; also strong extraction provider - **Perplexity** — direct answers with citations; via `PERPLEXITY_API_KEY` or `KILOCODE_API_KEY` - **You.com** — current-web / RAG-friendly snippets; also supports extraction - **SearXNG** — private/self-hosted search; no API key, just instance URL - **SerpBase** — low-cost Google SERP with prepaid credits; explicit/fallback-only (opt-in via `--provider serpbase` or add to `provider_priority` in `config.json`) ### Extraction providers `scripts/extract.py` auto-falls back across: 1. **Firecrawl** 2. **Linkup** 3. **Tavily** 4. **Exa** 5. **You.com** --- ## 🧠 Routing at a Glance Default priority (SerpBase excluded by design — opt-in only): ```text tavily → linkup → querit → exa → firecrawl → perplexity → brave → serper → you → searxng ``` Examples: ```bash python3 scripts/search.py -q "weather in Vienna today" # generic current-web intent → Brave or Serper python3 scripts/search.py -q "find credible sources for AI tutoring outcomes" # citation/evidence intent → Linkup python3 scripts/search.py -q "latest AI policy updates in Germany" # multilingual + recency → Querit or Tavily python3 scripts/search.py -p exa --exa-depth deep -q "LLM scaling laws research" python3 scripts/search.py -p firecrawl -q "YC startups web scraping" python3 scripts/search.py -p serpbase -q "best laptop 2026" # explicit SerpBase ``` Debug routing: ```bash python3 scripts/search.py --explain-routing -q "your query" ``` --- ## 📖 Extraction Examples ```bash python3 scripts/extract.py --url https://example.com python3 scripts/extract.py --url https://docs.linkup.so --provider linkup python3 scripts/extract.py --url https://example.com --url https://example.org --include-images python3 scripts/extract.py --url https://example.com --format html --include-raw-html ``` --- ## ⚙️ Configuration Notes - `.env.example` documents supported env vars - `config.example.json` includes provider priority and provider-specific defaults - `config.json` is your local runtime config - SearXNG still supports explicit URL config and docker-aware auto-detection - SerpBase is **explicit/fallback-only** by default; to include in auto-routing, append `"serpbase"` to `auto_routing.provider_priority` in `config.json` --- ## 🔒 Security **SearXNG SSRF protection:** - Enforces `http` / `https` only - Blocks common cloud metadata endpoints - Blocks private/internal IP resolution unless `SEARXNG_ALLOW_PRIVATE=1` - Uses operator-controlled config/env only for the instance URL --- ## ✅ Verification ```bash python3 -m unittest discover -s tests -p 'test_*.py' python3 scripts/search.py --explain-routing -q "find credible sources for climate change impacts" python3 scripts/extract.py --url https://example.com --provider auto --compact ```

path: CHANGELOG.md
content: # Changelog - Web Search Plus

## [3.1.0] - 2026-05-25

### Added
- **SerpBase provider integration** (closes [#4](https://github.com/robbyczgw-cla/web-search-plus/issues/4)) — low-cost Google SERP API with prepaid credits. Brings the skill to parity with `web-search-plus-plugin` v3.0.0.
  - New `search_serpbase()` function in `scripts/search.py` (Python port of plugin's canonical `searchSerpBase()` impl)
  - Explicit/fallback-only by design — NOT in default `provider_priority`. Opt-in via `--provider serpbase` OR append to `auto_routing.provider_priority` in `config.json`
  - New env var `SERPBASE_API_KEY` (https://serpbase.dev)
  - Treats HTTP 402/429 as transient (quota/rate-limit retry); business-error status fields surface as `ProviderRequestError`
  - SKILL.md + README + `.env.example` + `config.example.json` updated; package version bumped 3.0.3 → 3.1.0

## Unreleased

### Removed
- Removed the stale standalone homepage HTML from the OpenClaw Skill repository; `websearchplus.xyz` is maintained in the dedicated `websearchplus-xyz` homepage repository.

## [3.0.3] - 2026-05-06

### Fixed
- Version bump (previous version already published)

## [3.0.2] - 2026-05-06

### Fixed
- Fixed gzip/deflate/brotli decompression for HTTP responses across all providers
  - Added `_response_header()` and `_read_response_body()` helpers with Content-Encoding handling
  - Fixed 4 vulnerable `response.read().decode()` calls in `search.py` (make_request, make_get_request, search_youcom, search_searxng)
  - Fixed 1 vulnerable call in `extract.py` (request_json)
  - Removed forced `Accept-Encoding: gzip` header from Brave search (was causing guaranteed crash)
  - Added magic-number fallback (`b"\x1f\x8b"`) for mislabeled gzip responses
  - Brotli-encoded responses raise clear error instead of silent corruption

## [3.0.1] - 2026-05-03

### Fixed
- Version bump (previous version already published)

## [3.0.0] - 2026-05-03

### Added
- Added **Brave**, **Linkup**, and **Firecrawl** as first-class Python search providers, bringing provider parity to 10 search backends.
- Added `scripts/extract.py` for **URL extraction with automatic fallback** across Firecrawl, Linkup, Tavily, Exa, and You.com.
- Added targeted unit tests for routing parity, deterministic tie-breaking, extraction URL validation, and extraction fallback behavior.

### Changed
- Rebased `scripts/search.py` on newer Hermes parity logic while preserving this skill's Docker-aware SearXNG handling and local protections.
- Updated setup wizard, env template, config example, README, package metadata, and SKILL metadata for the 3.0.0 release.
- Default provider priority is now `tavily -> linkup -> querit -> exa -> firecrawl -> perplexity -> brave -> serper -> you -> searxng`.

### Notes
- Search parity is substantially improved with the OpenClaw plugin and Hermes port, but this skill still exposes the extraction flow as a companion script rather than a separate OpenClaw tool wrapper.

## [2.9.3] - 2026-04-20

### Added
- Docker-aware SearXNG auto-detection for zero-config local setups
- New `scripts/docker_detect.py` helper that detects containerized execution via `/.dockerenv`, `/proc/1/cgroup`, and `DOCKER`

### Changed
- `get_searxng_instance_url()` now falls back to auto-detected defaults when no explicit SearXNG URL is configured
- Uses `http://172.17.0.1:8080` inside Docker sandboxes and `http://127.0.0.1:8080` on the host for smoother local SearXNG access

### Release Notes
- Bumped skill/package metadata for the 2.9.3 ClawHub publish
- Based on merged GitHub PR #2: `feat: add Docker-aware SearXNG auto-detection`

## [2.9.2] - 2026-03-27

### Fixed
- Replaced hardcoded temporary cache path examples with portable `$TMP_DIR` placeholders in `TROUBLESHOOTING.md`

## [2.9.0] - 2026-03-12

### ✨ New Provider: Querit (Multilingual AI Search)

[Querit.ai](https://querit.ai) is a Singapore-based multilingual AI search API purpose-built for LLMs and RAG pipelines. 300 billion page index, 20+ countries, 10+ languages.

- Added **Querit** as the 7th search provider via `https://api.querit.ai/v1/search`
- Configure via `QUERIT_API_KEY` — optional, gracefully skipped if not set
- Routing score: `research * 0.65 + rag * 0.35 + recency * 0.45` — favored for multilingual and real-time queries
- Handles Querit's quirky `error_code=200` responses as success (not an error)
- Handles `IncompleteRead` as transient/retryable failure
- Live-tested with 10 benchmark queries ✅

### 🔧 Fixed: Fallback chain dies on unconfigured provider

- `sys.exit(1)` in `validate_api_key()` raised `SystemExit` (inherits from `BaseException`), which bypassed the `except Exception` fallback loop and killed the entire process instead of trying the next provider
- Replaced with catchable `ProviderConfigError` — fallback chain now continues correctly through all configured providers

### 🔧 Fixed: Perplexity citations are generic placeholders

- Previously extracted citation URLs via regex from the answer text, resulting in generic "Source 1" / "Source 2" labels
- Now uses the structured `data["citations"]` array from the Perplexity API response directly — results have readable titles
- Regex extraction kept as fallback when API doesn't return a `citations` field

### ✨ Improved: German locale routing patterns

- Added German-language signal patterns for local and news queries
- Improves auto-routing for queries like `"aktuelle Nachrichten"`, `"beste Restaurants Graz"`, `"KI Regulierung Europa"`

### 📝 Documentation

- Added Querit to README provider tables, routing examples, and API key setup section
- Added `querit_api_key` to `config.example.json`
- Updated `SKILL.md` provider mentions and env metadata
- Bumped package version to `2.9.0`


## [2.8.6] - 2026-03-03

### Changed
- Documented Perplexity Sonar Pro usage and refreshed release docs.


## [2.8.5] - 2026-02-20

### ✨ Feature: Perplexity freshness filter

- Added `freshness` parameter to Perplexity provider (`day`, `week`, `month`, `year`)
- Maps to Perplexity's native `search_recency_filter` parameter
- Example: `python3 scripts/search.py -p perplexity -q "latest AI news" --freshness day`
- Consistent with freshness support in Serper and Brave providers

## [2.8.4] - 2026-02-20

### 🔒 Security Fix: SSRF protection in setup wizard

- **Fixed:** `setup.py` SearXNG connection test had no SSRF protection (unlike `search.py`)
- **Before:** Operator could be tricked into probing internal networks during setup
- **After:** Same IP validation as `search.py` — blocks private IPs, cloud metadata, loopback
- **Credit:** ClawHub security scanner

## [2.8.3] - 2026-02-20

### 🐛 Critical Fix: Perplexity results empty

- **Fixed:** Perplexity provider returned 0 results because the AI-synthesized answer wasn't mapped into the results array
- **Before:** Only extracted URLs from the answer text were returned as results (often 0)
- **After:** The full answer is now the primary result (title, snippet with cleaned text), extracted source URLs follow as additional results
- **Impact:** Perplexity queries now always return at least 1 result with the synthesized answer

## [2.8.0] - 2026-02-20

### 🆕 New Provider: Perplexity (AI-Synthesized Answers)

Added Perplexity as the 6th search provider via Kilo Gateway — the first provider that returns **direct answers with citations** instead of just links:

#### Features
- **AI-Synthesized Answers**: Get a complete answer, not a list of links
- **Inline Citations**: Every claim backed by `[1][2][3]` source references
- **Real-Time Web Search**: Perplexity searches the web live, reads pages, and summarizes
- **Zero Extra Config**: Works through Kilo Gateway with your existing `KILOCODE_API_KEY`
- **Model**: `perplexity/sonar-pro` (best quality, supports complex queries)

#### Auto-Routing Signals
New direct-answer intent detection routes to Perplexity for:
- Status queries: "status of", "current state of", "what is the status"
- Local info: "events in [city]", "things to do in", "what's happening in"
- Direct questions: "what is", "who is", "when did", "how many"
- Current affairs: "this week", "this weekend", "right now", "today"

#### Usage Examples
```bash
# Auto-routed
python3 scripts/search.py -q "events in Graz Austria this weekend"  # → Perplexity
python3 scripts/search.py -q "what is the current status of Ethereum"  # → Perplexity

# Explicit
python3 scripts/search.py -p perplexity -q "latest AI regulation news"
```

#### Configuration
Requires `KILOCODE_API_KEY` environment variable (Kilo Gateway account).
No additional API key needed — Perplexity is accessed through Kilo's unified API.

```bash
export KILOCODE_API_KEY="your-kilo-key"
```

### 🔧 Routing Rebalance

Major overhaul of the auto-routing confidence scoring to fix Serper dominance:

#### Problem
Serper (Google) was winning ~90% of queries due to:
- High recency multiplier boosting Serper on any query with dates/years
- Default provider priority placing Serper first in ties
- Research and discovery signals not strong enough to override

#### Changes
- **Lowered Serper recency multiplier** — date mentions no longer auto-route to Google
- **Strengthened research signals** for Tavily:
  - Added: "status of", "what happened with", "how does X compare"
  - Boosted weights for comparison patterns (4.0 → 5.0)
- **Strengthened discovery signals** for Exa:
  - Added: "events in", "things to do in", "startups similar to"
  - Boosted weights for local discovery patterns
- **Updated provider priority order**: `tavily → exa → perplexity → serper → you → searxng`
  - Serper moved from 1st to 4th in tie-breaking
  - Research/discovery providers now win on ambiguous queries

#### Routing Test Results

| Query | Before | After | ✓ |
|-------|--------|-------|---|
| "latest OpenClaw version Feb 2026" | Serper | Serper | ✅ |
| "Ethereum Pectra upgrade status" | Serper | **Tavily** | ✅ |
| "events in Graz this weekend" | Serper | **Perplexity** | ✅ |
| "compare SearXNG vs Brave for AI agents" | Serper | **Tavily** | ✅ |
| "Sam Altman OpenAI news this week" | Serper | Serper | ✅ |
| "find startups similar to Kilo Code" | Serper | **Exa** | ✅ |

### 📊 Updated Provider Comparison

| Feature | Serper | Tavily | Exa | Perplexity | You.com | SearXNG |
|---------|:------:|:------:|:---:|:----------:|:-------:|:-------:|
| Speed | ⚡⚡⚡ | ⚡⚡ | ⚡⚡ | ⚡⚡ | ⚡⚡⚡ | ⚡ |
| Direct Answers | ✗ | ✗ | ✗ | ✓✓ | ✗ | ✗ |
| Citations | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |
| Local Events | ✓ | ✗ | ✓ | ✓✓ | ✗ | ✓ |
| Research | ✗ | ✓✓ | ✓ | ✓ | ✓ | ✗ |
| Discovery | ✗ | ✗ | ✓✓ | ✗ | ✗ | ✗ |
| Self-Hosted | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |

## [2.7.0] - 2026-02-14

### ✨ Added
- Provider cooldown tracking in `.cache/provider_health.json`
- Exponential cooldown on provider failures: **1m → 5m → 25m → 1h (cap)**
- Retry strategy for transient failures (timeout, 429, 503): up to 2 retries with backoff **1s → 3s → 9s**
- Smarter cache keys hashed from full request context (query/provider/max_results + locale, freshness, time_range, topic, search_engines, include_news, and related params)
- Cross-provider result deduplication by normalized URL during fallback merge

### 🔧 Changed
- Cooldown providers are skipped in routing while their cooldown is active
- Provider health is reset automatically after successful requests
- Fallback output now includes dedup metadata:
  - `deduplicated: true|false`
  - `metadata.dedup_count`


## [2.6.5] - 2026-02-11

### 🆕 File-Based Result Caching

Added local caching to save API costs on repeated searches:

#### Features
- **Automatic Caching**: Search results cached locally by default
- **1-Hour TTL**: Results expire after 3600 seconds (configurable)
- **Cache Indicators**: Response includes `cached: true/false` and `cache_age_seconds`
- **Zero-Cost Repeats**: Cached requests don't hit APIs

#### New CLI Options
- `--cache-ttl SECONDS` — Custom cache TTL (default: 3600)
- `--no-cache` — Bypass cache, always fetch fresh
- `--clear-cache` — Delete all cached results
- `--cache-stats` — Show cache statistics (entries, size, age)

#### Configuration
- **Cache directory**: `.cache/` in skill directory
- **Environment variable**: `WSP_CACHE_DIR` to override location
- **Cache key**: Based on query + provider + max_results (SHA256)

#### Usage Examples
```bash
# First request costs API credits
python3 scripts/search.py -q "AI startups"

# Second request is FREE (uses cache)
python3 scripts/search.py -q "AI startups"

# Force fresh results
python3 scripts/search.py -q "AI startups" --no-cache

# View stats
python3 scripts/search.py --cache-stats

# Clear everything
python3 scripts/search.py --clear-cache
```

#### Technical Details
- Cache files: JSON with metadata (_cache_timestamp, _cache_key, etc.)
- Automatic cleanup of expired entries on access
- Graceful handling of corrupted cache files

## [2.6.1] - 2026-02-04

- Privacy cleanup: removed hardcoded paths and personal info from docs

## [2.5.0] - 2026-02-03

### 🆕 New Provider: SearXNG (Privacy-First Meta-Search)

Added SearXNG as the 5th search provider, focused on privacy and self-hosted search:

#### Features
- **Privacy-Preserving**: No tracking, no profiling — your searches stay private
- **Multi-Source Aggregation**: Queries 70+ upstream engines (Google, Bing, DuckDuckGo, etc.)
- **$0 API Cost**: Self-hosted = unlimited queries with no API fees
- **Diverse Results**: Get perspectives from multiple search engines in one query
- **Customizable**: Choose which engines to use, set SafeSearch levels, language preferences

#### Auto-Routing Signals
New privacy/multi-source intent detection routes to SearXNG for:
- Privacy queries: "private", "anonymous", "without tracking", "no tracking"
- Multi-source: "aggregate results", "multiple sources", "diverse perspectives"
- Budget/free: "free search", "no api cost", "self-hosted search"
- German: "privat", "anonym", "ohne tracking", "verschiedene quellen"

#### Usage Examples
```bash
# Auto-routed
python3 scripts/search.py -q "search privately without tracking"  # → SearXNG

# Explicit
python3 scripts/search.py -p searxng -q "linux distros"
python3 scripts/search.py -p searxng -q "AI news" --engines "google,bing,duckduckgo"
python3 scripts/search.py -p searxng -q "privacy tools" --searxng-safesearch 2
```

#### Configuration
```json
{
  "searxng": {
    "instance_url": "https://your-instance.example.com",
    "safesearch": 0,
    "engines": null,
    "language": "en"
  }
}
```

#### Setup
SearXNG requires a self-hosted instance with JSON format enabled:
```bash
# Docker setup (5 minutes)
docker run -d -p 8080:8080 searxng/searxng

# Enable JSON in settings.yml:
# search:
#   formats: [html, json]

# Set instance URL
export SEARXNG_INSTANCE_URL="http://localhost:8080"
```

See: https://docs.searxng.org/admin/installation.html

### 📊 Updated Provider Comparison

| Feature | Serper | Tavily | Exa | You.com | SearXNG |
|---------|:------:|:------:|:---:|:-------:|:-------:|
| Privacy-First | ✗ | ✗ | ✗ | ✗ | ✓✓ |
| Self-Hosted | ✗ | ✗ | ✗ | ✗ | ✓ |
| API Cost | $$ | $$ | $$ | $ | **FREE** |
| Multi-Engine | ✗ | ✗ | ✗ | ✗ | ✓ (70+) |

### 🔧 Technical Changes

- Added `search_searxng()` function with full error handling
- Added `PRIVACY_SIGNALS` to QueryAnalyzer for auto-routing
- Updated setup wizard with SearXNG option (instance URL validation)
- Updated config.example.json with searxng section
- New CLI args: `--searxng-url`, `--searxng-safesearch`, `--engines`, `--categories`

---

## [2.4.4] - 2026-02-03

### 📝 Documentation: Provider Count Fix

- **Fixed:** "You can use 1, 2, or all 3" → "1, 2, 3, or all 4" (we have 4 providers now!)
- **Impact:** Accurate documentation for setup wizard

## [2.4.3] - 2026-02-03

### 📝 Documentation: Updated README

- **Added:** "NEW in v2.4.2" badge for You.com in SKILL.md
- **Impact:** ClawHub README now properly highlights You.com as new feature

## [2.4.2] - 2026-02-03

### 🐛 Critical Fix: You.com API Configuration

- **Fixed:** Incorrect hostname (`api.ydc-index.io` → `ydc-index.io`)
- **Fixed:** Incorrect header name (`X-API-Key` → `X-API-KEY` uppercase)
- **Impact:** You.com now works correctly - was giving 403 Forbidden before
- **Status:** ✅ Fully tested and working

## [2.4.1] - 2026-02-03

### 🐛 Bugfix: You.com URL Encoding

- **Fixed:** URL encoding for You.com queries - spaces and special characters now properly encoded
- **Impact:** Queries with spaces (e.g., "OpenClaw AI framework") work correctly now
- **Technical:** Added `urllib.parse.quote` for parameter encoding

## [2.4.0] - 2026-02-03

### 🆕 New Provider: You.com

Added You.com as the 4th search provider, optimized for RAG applications and real-time information:

#### Features
- **LLM-Ready Snippets**: Pre-extracted, query-aware text excerpts perfect for feeding into AI models
- **Unified Web + News**: Get both web pages and news articles in a single API call
- **Live Crawling**: Fetch full page content on-demand in Markdown format (`--livecrawl`)
- **Automatic News Classification**: Intelligently includes news results based on query intent
- **Freshness Controls**: Filter by recency (day, week, month, year, or date range)
- **SafeSearch Support**: Content filtering (off, moderate, strict)

#### Auto-Routing Signals
New RAG/Real-time intent detection routes to You.com for:
- RAG context queries: "summarize", "key points", "tldr", "context for"
- Real-time info: "latest news", "current status", "right now", "what's happening"
- Information synthesis: "updates on", "situation", "main takeaways"

#### Usage Examples
```bash
# Auto-routed
python3 scripts/search.py -q "summarize key points about AI regulation"  # → You.com

# Explicit
python3 scripts/search.py -p you -q "climate change" --livecrawl all
python3 scripts/search.py -p you -q "tech news" --freshness week
```

#### Configuration
```json
{
  "you": {
    "country": "US",
    "language": "en",
    "safesearch": "moderate",
    "include_news": true
  }
}
```

#### API Key Setup
```bash
export YOU_API_KEY="your-key"  # Get from https://api.you.com
```

### 📊 Updated Provider Comparison

| Feature | Serper | Tavily | Exa | You.com |
|---------|:------:|:------:|:---:|:-------:|
| Speed | ⚡⚡⚡ | ⚡⚡ | ⚡⚡ | ⚡⚡⚡ |
| News Integration | ✓ | ✗ | ✗ | ✓ |
| RAG-Optimized | ✗ | ✓ | ✗ | ✓✓ |
| Full Page Content | ✗ | ✓ | ✓ | ✓ |

---

## [2.1.5] - 2026-01-27

### 📝 Documentation

- Added warning about NOT using Tavily/Serper/Exa in core OpenClaw config
- Core OpenClaw only supports `brave` as the built-in provider
- This skill's providers must be used via environment variables and scripts, not `openclaw.json`

## [2.1.0] - 2026-01-23

### 🧠 Intelligent Multi-Signal Routing

Completely overhauled auto-routing with sophisticated query analysis:

#### Intent Classification
- **Shopping Intent**: Detects price patterns ("how much", "cost of"), purchase signals ("buy", "order"), deal keywords, and product+brand combinations
- **Research Intent**: Identifies explanation patterns ("how does", "why does"), analysis signals ("pros and cons", "compare"), learning keywords, and complex multi-clause queries
- **Discovery Intent**: Recognizes similarity patterns ("similar to", "alternatives"), company discovery signals, URL/domain detection, and academic patterns

#### Linguistic Pattern Detection
- "How much" / "price of" → Shopping (Serper)
- "How does" / "Why does" / "Explain" → Research (Tavily)
- "Companies like" / "Similar to" / "Alternatives" → Discovery (Exa)
- Product + Brand name combos → Shopping (Serper)
- URLs and domains in query → Similar search (Exa)

#### Query Analysis Features
- **Complexity scoring**: Long, multi-clause queries get routed to research providers
- **URL detection**: Automatic detection of URLs/domains triggers Exa similar search
- **Brand recognition**: Tech brands (Apple, Samsung, Sony, etc.) with product terms → shopping
- **Recency signals**: "latest", "2026", "breaking" boost news mode

#### Confidence Scoring
- **HIGH (70-100%)**: Strong signal match, very reliable routing
- **MEDIUM (40-69%)**: Good match, should work well
- **LOW (0-39%)**: Ambiguous query, using fallback provider
- Confidence based on absolute signal strength + relative margin over alternatives

#### Enhanced Debug Mode
```bash
python3 scripts/search.py --explain-routing -q "your query"
```

Now shows:
- Routing decision with confidence level
- All provider scores
- Top matched signals with weights
- Query analysis (complexity, URL detection, recency focus)
- All matched patterns per provider

### 🔧 Technical Changes

#### QueryAnalyzer Class
New `QueryAnalyzer` class with:
- `SHOPPING_SIGNALS`: 25+ weighted patterns for shopping intent
- `RESEARCH_SIGNALS`: 30+ weighted patterns for research intent
- `DISCOVERY_SIGNALS`: 20+ weighted patterns for discovery intent
- `LOCAL_NEWS_SIGNALS`: 25+ patterns for local/news queries
- `BRAND_PATTERNS`: Tech brand detection regex

#### Signal Weighting
- Multi-word phrases get higher weights (e.g., "how much" = 4.0 vs "price" = 3.0)
- Strong signals: price patterns (4.0), similarity patterns (5.0), URLs (5.0)
- Medium signals: product terms (2.5), learning keywords (2.5)
- Bonus scoring: Product+brand combo (+3.0), complex query (+2.5)

#### Improved Output Format
```json
{
  "routing": {
    "auto_routed": true,
    "provider": "serper",
    "confidence": 0.78,
    "confidence_level": "high",
    "reason": "high_confidence_match",
    "top_signals": [{"matched": "price", "weight": 3.0}],
    "scores": {"serper": 7.0, "tavily": 0.0, "exa": 0.0}
  }
}
```

### 📚 Documentation Updates

- **SKILL.md**: Complete rewrite with signal tables and confidence scoring guide
- **README.md**: Updated with intelligent routing examples and confidence levels
- **FAQ**: Updated to explain multi-signal analysis

### 🧪 Test Results

| Query | Provider | Confidence | Signals |
|-------|----------|------------|---------|
| "how much does iPhone 16 cost" | Serper | 68% | "how much", brand+product |
| "how does quantum entanglement work" | Tavily | 86% HIGH | "how does", "what are", "implications" |
| "startups similar to Notion" | Exa | 76% HIGH | "similar to", "Series A" |
| "companies like stripe.com" | Exa | 100% HIGH | URL detected, "companies like" |
| "MacBook Pro M3 specs review" | Serper | 70% HIGH | brand+product, "specs", "review" |
| "Tesla" | Serper | 0% LOW | No signals (fallback) |
| "arxiv papers on transformers" | Exa | 58% | "arxiv" |
| "latest AI news 2026" | Serper | 77% HIGH | "latest", "news", "2026" |

---

## [2.0.0] - 2026-01-23

### 🎉 Major Features

#### Smart Auto-Routing
- **Automatic provider selection** based on query analysis
- No need to manually choose provider - just search!
- Intelligent keyword matching for routing decisions
- Pattern detection for query types (shopping, research, discovery)
- Scoring system for provider selection

#### User Configuration
- **config.json**: Full control over auto-routing behavior
- **Configurable keyword mappings**: Add your own routing keywords
- **Provider priority**: Set tie-breaker order
- **Disable providers**: Turn off providers you don't have API keys for
- **Enable/disable auto-routing**: Opt-in or opt-out as needed

#### Debugging Tools
- **--explain-routing** flag: See exactly why a provider was selected
- Detailed routing metadata in JSON responses
- Shows matched keywords and routing scores

### 📚 Documentation

- **README.md**: Complete auto-routing guide with examples
- **SKILL.md**: Detailed routing logic and configuration reference
- **FAQ section**: Common questions about auto-routing
- **Configuration examples**: Pre-built configs for common use cases

---

## [1.0.x] - Initial Release

- Multi-provider search: Serper, Tavily, Exa
- Manual provider selection with `-p` flag
- Unified JSON output format
- Provider-specific options (--depth, --category, --similar-url, etc.)
- Domain filtering for Tavily/Exa
- Date filtering for Exa


path: FAQ.md
content: # Frequently Asked Questions

## General

### What is Web Search Plus 3.0.0?

A Python OpenClaw skill for unified web search plus URL extraction.

It supports 10 search providers:

- Serper
- Brave
- Tavily
- Querit
- Linkup
- Exa
- Firecrawl
- Perplexity via direct API or Kilo gateway
- You.com
- SearXNG

It also adds URL extraction through `scripts/extract.py` with fallback across:

- Firecrawl
- Linkup
- Tavily
- Exa
- You.com

### Is this the same as the OpenClaw plugin?

No.

- The **plugin** registers native OpenClaw tools like `web_search_plus` and `web_extract_plus`.
- The **skill** provides scripts and instructions for agent workflows: `scripts/search.py` and `scripts/extract.py`.

The 3.0.0 skill release brings the old skill much closer to plugin/Hermes provider parity, but the plugin is still the cleaner native OpenClaw route for tool registration.

### Which should I use?

Use the plugin for new OpenClaw setups.

Use this skill when you want:

- portable scripts
- manual CLI control
- an inspectable Python implementation
- skill-style instructions for agents
- compatibility with older workflows that already call `scripts/search.py`

## Setup

### Which API keys do I need?

Only one search provider is required to start.

Any one of these is enough for search:

- `SERPER_API_KEY`
- `BRAVE_API_KEY`
- `TAVILY_API_KEY`
- `QUERIT_API_KEY`
- `LINKUP_API_KEY`
- `EXA_API_KEY`
- `FIRECRAWL_API_KEY`
- `PERPLEXITY_API_KEY`
- `KILOCODE_API_KEY`
- `YOU_API_KEY`
- `SEARXNG_INSTANCE_URL`

Extraction needs one of:

- `FIRECRAWL_API_KEY`
- `LINKUP_API_KEY`
- `TAVILY_API_KEY`
- `EXA_API_KEY`
- `YOU_API_KEY`

### Where do I get keys?

- Serper: <https://serper.dev>
- Brave: <https://brave.com/search/api/>
- Tavily: <https://tavily.com>
- Querit: <https://querit.ai>
- Linkup: <https://linkup.so>
- Exa: <https://exa.ai>
- Firecrawl: <https://firecrawl.dev>
- Perplexity: <https://www.perplexity.ai/settings/api>
- Kilo gateway: <https://kilo.ai>
- You.com: <https://api.you.com>
- SearXNG: <https://docs.searxng.org/admin/installation.html>

### How do I configure keys?

Use `.env`:

```bash
cp .env.example .env
# edit .env
```

Or use `config.json` for provider-specific settings.

Priority order for credentials is generally:

- `config.json`
- `.env`
- process environment

## Routing

### How does auto-routing decide?

The skill scores query signals and chooses among configured providers only.

Typical routing:

- shopping, product, local, broad Google-style web → Serper or Brave
- generic current web → Brave or Serper
- research/explanation → Tavily
- source/citation/evidence queries → Linkup
- multilingual/international updates → Querit or Tavily
- semantic discovery, similar sites, papers → Exa
- scrape-ready discovery → Firecrawl
- direct answer / cited summary → Perplexity via direct API or Kilo gateway
- RAG/current-web snippets → You.com
- private/self-hosted search → SearXNG

### How do I see the routing decision?

```bash
python3 scripts/search.py --explain-routing -q "your query"
```

### What if it picks the wrong provider?

Force a provider:

```bash
python3 scripts/search.py -p linkup -q "credible sources for AI tutoring outcomes"
python3 scripts/search.py -p firecrawl -q "YC startups web scraping"
```

Or adjust `auto_routing.provider_priority` / `disabled_providers` in `config.json`.

### What does low confidence mean?

The query did not strongly match one provider. The skill may fall back to the configured fallback provider, usually Serper.

## Extraction

### How do I extract URL content?

```bash
python3 scripts/extract.py --url https://example.com
```

Multiple URLs:

```bash
python3 scripts/extract.py --url https://example.com --url https://example.org
```

Force a provider:

```bash
python3 scripts/extract.py --provider firecrawl --url https://example.com
```

### Which extraction provider is tried first?

Auto extraction tries:

- Firecrawl
- Linkup
- Tavily
- Exa
- You.com

Missing credentials are skipped. Failed providers fall through to the next configured provider.

### Can extraction return HTML?

Yes, when the provider supports it:

```bash
python3 scripts/extract.py --url https://example.com --format html --include-raw-html
```

## Caching

### How does caching work?

Search results are cached locally by query, provider, result count, and relevant params.

Default TTL: 3600 seconds.

### Where are cached results stored?

In `.cache/` inside the skill folder by default.

Override with:

```bash
export WSP_CACHE_DIR="/path/to/custom/cache"
```

### How do I inspect or clear cache?

```bash
python3 scripts/search.py --cache-stats
python3 scripts/search.py --clear-cache
```

### How do I skip cache?

```bash
python3 scripts/search.py -q "query" --no-cache
```

## SearXNG

### Do I need my own SearXNG instance?

Usually yes. Most public SearXNG instances disable JSON API.

### Is SearXNG free?

Yes, the software is free and self-hosted. You only pay for hosting if you run it on a VPS.

### Is private-network access allowed?

Blocked by default for safety. Only set `SEARXNG_ALLOW_PRIVATE=1` when you intentionally use a trusted private SearXNG instance.

## Production use

### Is this production-ready?

Yes, with normal API caveats:

- automatic fallback
- rate-limit handling
- provider cooldowns
- local cache
- SSRF protections for SearXNG
- configurable provider priority

For native OpenClaw tool usage, prefer the plugin. For script-based skill workflows, this skill is ready once tests pass.

### What if a provider is rate-limited?

The skill tries fallback providers when available. You can also temporarily disable exhausted providers in `config.json`.

## Updating

### How do I update?

Via ClawHub:

```bash
clawhub update web-search-plus --registry "https://www.clawhub.ai" --no-input
```

Manual git workflow:

```bash
cd /path/to/skills/web-search-plus
git pull origin main
python3 scripts/setup.py
```


path: README.md
content: # Web Search Plus

<p align="center">
  <img src="docs/assets/web-search-plus-logo.png" alt="web search plus logo" width="180">
</p>

Unified multi-provider web search and URL extraction for OpenClaw-style agent workflows.

Current version: **3.1.0**

## What changed in 3.1.0

- Add **SerpBase** as the 11th search provider — low-cost Google SERP API with prepaid credits (https://serpbase.dev)
- Explicit/fallback-only by design: NOT in default auto-routing priority. Use `--provider serpbase` or opt-in via `config.json`.
- Closes [#4](https://github.com/robbyczgw-cla/web-search-plus/issues/4); brings skill to parity with `web-search-plus-plugin` v3.0.0

## What changed in 3.0.x

- Add **Brave**, **Linkup**, and **Firecrawl** search providers to the Python skill
- Add **URL extraction** via `scripts/extract.py` with auto fallback across Firecrawl, Linkup, Tavily, Exa, and You.com
- Align routing/fallback behavior and docs more closely with the OpenClaw plugin and Hermes port
- Keep existing **Exa deep** / **deep-reasoning**, cooldown, retry, cache, and SearXNG protections

## Search providers

- **Serper** — shopping, local, broad Google-style web results
- **Brave** — independent general web index, good broad fallback
- **Tavily** — research and explanation queries
- **Querit** — multilingual and international AI search
- **Linkup** — citation/source-grounded search
- **Exa** — semantic discovery and deep synthesis
- **Firecrawl** — search with scrape-ready metadata
- **Perplexity via Kilo** — answer-first web results
- **You.com** — current-web / RAG-ish queries
- **SearXNG** — privacy-first self-hosted metasearch
- **SerpBase** — low-cost Google SERP, prepaid credits, **explicit/fallback-only**

## Extraction providers

`scripts/extract.py` supports:

- Firecrawl
- Linkup
- Tavily
- Exa
- You.com

Auto extraction tries them in that order and falls back when a provider is unconfigured or fails.

## Quick start

```bash
cp .env.example .env
# fill in at least one key or SEARXNG_INSTANCE_URL

python3 scripts/search.py -q "latest OpenClaw release"
python3 scripts/extract.py --url https://example.com
```

Or use the interactive wizard:

```bash
python3 scripts/setup.py
```

## Search examples

```bash
python3 scripts/search.py -q "weather in Vienna today"
# auto-routes to Brave or Serper for broad current-web intent

python3 scripts/search.py -q "find credible sources for AI tutoring outcomes"
# auto-routes to Linkup when available

python3 scripts/search.py -q "latest AI policy updates in Germany"
# often Querit / Tavily depending on configured providers

python3 scripts/search.py -p exa --exa-depth deep -q "LLM scaling laws research"
python3 scripts/search.py -p firecrawl -q "YC startups web scraping"
python3 scripts/search.py -p serpbase -q "best laptop 2026"
# explicit SerpBase call — not used by auto-routing unless added to provider_priority
```

## Extraction examples

```bash
python3 scripts/extract.py --url https://example.com
python3 scripts/extract.py --url https://docs.linkup.so --provider linkup
python3 scripts/extract.py --url https://example.com --url https://example.org --include-images
```

## Routing notes

Provider priority now defaults to:

```text
tavily -> linkup -> querit -> exa -> firecrawl -> perplexity -> brave -> serper -> you -> searxng
```

Notable behavior:

- Brave and Serper share generic web/current-info intent and use deterministic tie-breaking
- Linkup gets explicit boosts for citation/source/evidence-style queries
- Firecrawl can win discovery/research-ish queries when configured
- Exa can auto-upgrade to `deep` or `deep-reasoning` based on query signals
- Failing or cooling-down providers are skipped by fallback routing

## Config files

- `.env.example` — provider credentials template
- `config.example.json` — routing and provider settings template
- `config.json` — your live local config (created/edited locally)

## Verification

Suggested local checks:

```bash
python3 -m unittest discover -s tests -p 'test_*.py'
python3 scripts/search.py --explain-routing -q "find credible sources for climate change impacts"
python3 scripts/extract.py --url https://example.com --provider auto --compact
```

## Related references

- OpenClaw plugin: `../projects/web-search-plus-plugin`
- Hermes port: `../projects/hermes-web-search-plus`


path: TROUBLESHOOTING.md
content: # Troubleshooting Guide

## Caching Issues (v2.7.0+)

### Cache not working / always fetching fresh

**Symptoms:**
- Every request hits the API
- `"cached": false` even for repeated queries

**Solutions:**
1. Check cache directory exists and is writable:
   ```bash
   ls -la .cache/  # Should exist in skill directory
   ```
2. Verify `--no-cache` isn't being passed
3. Check disk space isn't full
4. Ensure query is EXACTLY the same (including provider and max_results)

### Stale results from cache

**Symptoms:**
- Getting outdated information
- Cache TTL seems too long

**Solutions:**
1. Use `--no-cache` to force fresh results
2. Reduce TTL: `--cache-ttl 1800` (30 minutes)
3. Clear cache: `python3 scripts/search.py --clear-cache`

### Cache growing too large

**Symptoms:**
- Disk space filling up
- Many .json files in `.cache/`

**Solutions:**
1. Clear cache periodically:
   ```bash
   python3 scripts/search.py --clear-cache
   ```
2. Set up a cron job to clear weekly
3. Use a smaller TTL so entries expire faster

### "Permission denied" when caching

**Symptoms:**
- Cache write errors in stderr
- Searches work but don't cache

**Solutions:**
1. Check directory permissions: `chmod 755 .cache/`
2. Use custom cache dir: `export WSP_CACHE_DIR="$TMP_DIR/wsp-cache"`

---

## Common Issues

### "No API key found" error

**Symptoms:**
```
Error: No API key found for serper
```

**Solutions:**
1. Check `.env` exists in skill folder with `export VAR=value` format
2. Keys auto-load from skill's `.env` since v2.2.0
3. Or set in system environment: `export SERPER_API_KEY="..."`
4. Verify key format in config.json:
   ```json
   { "serper": { "api_key": "your-key" } }
   ```

**Priority order:** config.json > .env > environment variable

---

### Getting empty results

**Symptoms:**
- Search returns no results
- `"results": []` in JSON output

**Solutions:**
1. Check API key is valid (try the provider's web dashboard)
2. Try a different provider with `-p`
3. Some queries have no results (very niche topics)
4. Check if provider is rate-limited
5. Verify internet connectivity

**Debug:**
```bash
python3 scripts/search.py -q "test query" --verbose
```

---

### Rate limited

**Symptoms:**
```
Error: 429 Too Many Requests
Error: Rate limit exceeded
```

**Good news:** Since v2.2.5, automatic fallback kicks in! If one provider hits rate limits, the script automatically tries the next provider.

**Solutions:**
1. Wait for rate limit to reset (usually 1 hour or end of day)
2. Use a different provider: `-p tavily` instead of `-p serper`
3. Check free tier limits:
   - Serper: 2,500 free total
   - Tavily: 1,000/month free
   - Exa: 1,000/month free
4. Upgrade to paid tier for higher limits
5. Use SearXNG (self-hosted, unlimited)

**Fallback info:** Response will include `routing.fallback_used: true` when fallback was used.

---

### SearXNG: "403 Forbidden"

**Symptoms:**
```
Error: 403 Forbidden
Error: JSON format not allowed
```

**Cause:** Most public SearXNG instances disable JSON API to prevent bot abuse.

**Solution:** Self-host your own instance:
```bash
docker run -d -p 8080:8080 searxng/searxng
```

Then enable JSON in `settings.yml`:
```yaml
search:
  formats:
    - html
    - json  # Add this!
```

Restart the container and update your config:
```json
{
  "searxng": {
    "instance_url": "http://localhost:8080"
  }
}
```

---

### SearXNG: Slow responses

**Symptoms:**
- SearXNG takes 2-5 seconds
- Other providers are faster

**Explanation:** This is expected behavior. SearXNG queries 70+ upstream engines in parallel, which takes longer than direct API calls.

**Trade-off:** Slower but privacy-preserving + multi-source + $0 cost.

**Solutions:**
1. Accept the trade-off for privacy benefits
2. Limit engines for faster results:
   ```bash
   python3 scripts/search.py -p searxng -q "query" --engines "google,bing"
   ```
3. Use SearXNG as fallback (put last in priority list)

---

### Auto-routing picks wrong provider

**Symptoms:**
- Query about research goes to Serper
- Query about shopping goes to Tavily

**Debug:**
```bash
python3 scripts/search.py --explain-routing -q "your query"
```

This shows the full analysis:
```json
{
  "query": "how much does iPhone 16 Pro cost",
  "routing_decision": {
    "provider": "serper",
    "confidence": 0.68,
    "reason": "moderate_confidence_match"
  },
  "scores": {"serper": 7.0, "tavily": 0.0, "exa": 0.0},
  "top_signals": [
    {"matched": "how much", "weight": 4.0},
    {"matched": "brand + product detected", "weight": 3.0}
  ]
}
```

**Solutions:**
1. Override with explicit provider: `-p tavily`
2. Rephrase query to be more explicit about intent
3. Adjust `confidence_threshold` in config.json (default: 0.3)

---

### Config not loading

**Symptoms:**
- Changes to config.json not applied
- Using default values instead

**Solutions:**
1. Check JSON syntax (use a validator)
2. Ensure file is in skill directory: `/path/to/skills/web-search-plus/config.json`
3. Check file permissions
4. Run setup wizard to regenerate:
   ```bash
   python3 scripts/setup.py --reset
   ```

**Validate JSON:**
```bash
python3 -m json.tool config.json
```

---

### Python dependencies missing

**Symptoms:**
```
ModuleNotFoundError: No module named 'requests'
```

**Solution:**
```bash
pip3 install requests
```

Or install all dependencies:
```bash
pip3 install -r requirements.txt
```

---

### Timeout errors

**Symptoms:**
```
Error: Request timeout after 30s
```

**Causes:**
- Slow network connection
- Provider API issues
- SearXNG instance overloaded

**Solutions:**
1. Try again (temporary issue)
2. Switch provider: `-p serper`
3. Check your internet connection
4. If using SearXNG, check instance health

---

### Duplicate results

**Symptoms:**
- Same result appears multiple times
- Results overlap between providers

**Solution:** This is expected when using auto-fallback or multiple providers. The skill doesn't deduplicate across providers.

For single-provider results:
```bash
python3 scripts/search.py -p serper -q "query"
```

---

## Debug Mode

For detailed debugging:

```bash
# Verbose output
python3 scripts/search.py -q "query" --verbose

# Show routing decision
python3 scripts/search.py -q "query" --explain-routing

# Dry run (no actual search)
python3 scripts/search.py -q "query" --dry-run

# Test specific provider
python3 scripts/search.py -p tavily -q "query" --verbose
```

---

## Getting Help

**Still stuck?**

1. Check the full documentation in `README.md`
2. Run the setup wizard: `python3 scripts/setup.py`
3. Review `FAQ.md` for common questions
4. Open an issue: https://github.com/robbyczgw-cla/web-search-plus/issues


path: config.example.json
content: {
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$comment": "Web Search Plus v3.1.0 example config — search routing plus extraction-capable provider settings (incl. opt-in SerpBase)",
  "defaults": {
    "provider": "serper",
    "max_results": 5
  },
  "auto_routing": {
    "enabled": true,
    "fallback_provider": "serper",
    "provider_priority": [
      "tavily",
      "linkup",
      "querit",
      "exa",
      "firecrawl",
      "perplexity",
      "brave",
      "serper",
      "you",
      "searxng"
    ],
    "disabled_providers": [],
    "confidence_threshold": 0.3
  },
  "serper": {
    "country": "us",
    "language": "en",
    "type": "search"
  },
  "brave": {
    "country": "US",
    "search_lang": "en",
    "safesearch": "moderate"
  },
  "tavily": {
    "depth": "advanced",
    "topic": "general"
  },
  "querit": {
    "base_url": "https://api.querit.ai",
    "base_path": "/v1/search",
    "timeout": 10
  },
  "linkup": {
    "api_url": "https://api.linkup.so/v1/search",
    "depth": "standard",
    "output_type": "searchResults",
    "timeout": 30
  },
  "exa": {
    "type": "neural",
    "depth": "normal",
    "verbosity": "standard"
  },
  "firecrawl": {
    "api_url": "https://api.firecrawl.dev/v2/search",
    "country": "US",
    "timeout": 30000,
    "sources": ["web"],
    "ignore_invalid_urls": false
  },
  "perplexity": {
    "api_url": "https://api.kilo.ai/api/gateway/chat/completions",
    "model": "perplexity/sonar-pro"
  },
  "you": {
    "country": "us",
    "safesearch": "moderate"
  },
  "searxng": {
    "instance_url": null,
    "safesearch": 0,
    "engines": null,
    "language": "en"
  },
  "serpbase": {
    "$comment": "SerpBase is explicit/fallback-only. To include in auto-routing, append 'serpbase' to auto_routing.provider_priority.",
    "api_url": "https://api.serpbase.com/search",
    "timeout": 30
  }
}


path: package.json
content: {
  "name": "web-search-plus",
  "version": "3.1.0",
  "description": "Unified multi-provider web search and URL extraction skill with intelligent auto-routing across Serper, Brave, Tavily, Querit, Linkup, Exa, Firecrawl, Perplexity, You.com, SearXNG, and SerpBase.",
  "keywords": [
    "openclaw",
    "skill",
    "search",
    "web-search",
    "serper",
    "brave",
    "tavily",
    "linkup",
    "exa",
    "firecrawl",
    "you",
    "you.com",
    "serpbase",
    "google-search",
    "research",
    "semantic-search",
    "ai-agent",
    "auto-routing",
    "smart-routing",
    "multi-provider",
    "shopping",
    "product-search",
    "similar-sites",
    "company-discovery",
    "rag",
    "real-time",
    "free-tier",
    "api-aggregator",
    "querit",
    "multilingual-search",
    "web-extract",
    "content-extraction",
    "firecrawl",
    "brave-search",
    "linkup"
  ],
  "author": "robbyczgw-cla",
  "license": "MIT",
  "repository": {
    "type": "git",
    "url": "https://github.com/robbyczgw-cla/web-search-plus.git"
  },
  "homepage": "https://clawhub.ai/robbyczgw-cla/web-search-plus",
  "bugs": {
    "url": "https://github.com/robbyczgw-cla/web-search-plus/issues"
  },
  "files": [
    "SKILL.md",
    "README.md",
    "scripts/",
    ".env.example",
    "config.example.json",
    "CHANGELOG.md"
  ],
  "clawhub": {
    "skill": true,
    "triggers": [
      "search",
      "find",
      "look up",
      "research"
    ],
    "capabilities": [
      "web-search",
      "image-search",
      "semantic-search",
      "multi-provider",
      "web-extract"
    ],
    "requirements": {
      "bins": [
        "python3",
        "bash"
      ],
      "env": {
        "SERPER_API_KEY": "optional",
        "BRAVE_API_KEY": "optional",
        "TAVILY_API_KEY": "optional",
        "QUERIT_API_KEY": "optional",
        "LINKUP_API_KEY": "optional",
        "EXA_API_KEY": "optional",
        "FIRECRAWL_API_KEY": "optional",
        "PERPLEXITY_API_KEY": "optional \u2014 direct Perplexity provider credential",
        "KILOCODE_API_KEY": "optional \u2014 alternative Perplexity provider via Kilo Gateway",
        "YOU_API_KEY": "optional",
        "SEARXNG_INSTANCE_URL": "optional"
      }
    }
  }
}

path: scripts/docker_detect.py
content: import os
import sys
from pathlib import Path


def is_docker() -> bool:
    """Detect if running inside a Docker container.
    
    Checks multiple signals:
    - /.dockerenv file
    - /proc/1/cgroup containing 'docker' or 'kubepods'
    - DOCKER environment variable
    """
    if os.path.exists("/.dockerenv"):
        return True
    
    try:
        with open("/proc/1/cgroup", "r") as f:
            cgroup = f.read().lower()
            if "docker" in cgroup or "kubepods" in cgroup:
                return True
    except (FileNotFoundError, PermissionError):
        pass
    
    if os.environ.get("DOCKER"):
        return True
    
    return False


def get_searxng_url() -> str:
    """Get the appropriate SearXNG URL based on environment.
    
    Returns:
        - http://172.17.0.1:8080 if running in Docker
        - http://127.0.0.1:8080 if running on host
    """
    if is_docker():
        return "http://172.17.0.1:8080"
    return "http://127.0.0.1:8080"


def ensure_searxng_env():
    """Set SEARXNG_INSTANCE_URL environment variable if not already set."""
    if "SEARXNG_INSTANCE_URL" not in os.environ:
        os.environ["SEARXNG_INSTANCE_URL"] = get_searxng_url()


if __name__ == "__main__":
    print(f"In Docker: {is_docker()}")
    print(f"SearXNG URL: {get_searxng_url()}")

path: scripts/extract.py
content: #!/usr/bin/env python3
"""
Web Search Plus — URL extraction with automatic provider fallback.
Supports: Firecrawl, Linkup, Tavily, Exa, You.com

Usage:
    python3 scripts/extract.py --url https://example.com
    python3 scripts/extract.py --url https://example.com --provider firecrawl --format markdown
    python3 scripts/extract.py --url https://example.com --url https://example.org --include-images
"""

import argparse
import gzip
import json
import os
import sys
import zlib
from pathlib import Path
from typing import Any, Dict, List, Optional
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen

EXTRACT_PROVIDER_PRIORITY = ["firecrawl", "linkup", "tavily", "exa", "you"]


def _load_env_file() -> None:
    env_paths = [Path(__file__).parent.parent / ".env", Path(__file__).parent / ".env"]
    for env_path in env_paths:
        if not env_path.exists():
            continue
        with open(env_path, encoding="utf-8") as handle:
            for line in handle:
                line = line.strip()
                if line and not line.startswith("#") and "=" in line:
                    if line.startswith("export "):
                        line = line[7:]
                    key, _, value = line.partition("=")
                    key = key.strip()
                    value = value.strip().strip('"').strip("'")
                    if key and key not in os.environ:
                        os.environ[key] = value


_load_env_file()


def _response_header(response, name: str, default: str = "") -> str:
    """Get response header safely (works with HTTPError too)."""
    try:
        val = response.getheader(name) if hasattr(response, "getheader") else response.headers.get(name)
        return val or default
    except Exception:
        return default


def _read_response_body(response) -> bytes:
    """Read response body with gzip/deflate decompression support."""
    raw = response.read()
    encoding = _response_header(response, "Content-Encoding", "").lower().strip()

    if encoding in ("gzip", "x-gzip"):
        try:
            return gzip.decompress(raw)
        except Exception:
            if raw.startswith(b"\x1f\x8b"):
                return gzip.decompress(raw)
            return raw
    elif encoding == "deflate":
        try:
            return zlib.decompress(raw, -zlib.MAX_WBITS)
        except zlib.error:
            try:
                return zlib.decompress(raw)
            except zlib.error:
                return raw
    elif encoding == "br":
        try:
            import brotli
            return brotli.decompress(raw)
        except ImportError:
            raise RuntimeError(
                "Brotli-encoded response received but brotli library not installed. Install with: pip install brotli"
            )
        except Exception:
            raise RuntimeError("Failed to decompress brotli-encoded response")

    return raw


def request_json(url: str, init: Dict[str, Any], timeout: int = 30) -> Any:
    body = init.get("body")
    data = body.encode("utf-8") if isinstance(body, str) else body
    req = Request(url, data=data, method=init.get("method", "GET"))
    for key, value in (init.get("headers") or {}).items():
        req.add_header(key, value)
    try:
        with urlopen(req, timeout=max(1, timeout)) as response:
            text = _read_response_body(response).decode("utf-8")
            return json.loads(text) if text else {}
    except HTTPError as exc:
        payload = exc.read().decode("utf-8", errors="replace") if hasattr(exc, "read") else ""
        try:
            data = json.loads(payload) if payload else {}
        except json.JSONDecodeError:
            data = {}
        message = data.get("error") or data.get("message") or data.get("detail") or data.get("warning") or f"HTTP {exc.code}"
        raise RuntimeError(str(message)) from exc
    except URLError as exc:
        raise RuntimeError(str(exc.reason)) from exc


def title_from_url(url: str) -> str:
    try:
        from urllib.parse import urlparse
        parsed = urlparse(url)
        last_segment = [part for part in parsed.path.split("/") if part]
        return (last_segment[-1] if last_segment else parsed.hostname) or url
    except Exception:
        return url


def normalize_images(images: Any) -> Optional[List[Dict[str, str]]]:
    if not isinstance(images, list):
        return None
    normalized = []
    for image in images:
        if not image:
            continue
        if isinstance(image, str):
            normalized.append({"url": image})
        elif isinstance(image, dict) and isinstance(image.get("url"), str) and image.get("url"):
            item = {"url": image["url"]}
            if isinstance(image.get("alt"), str) and image.get("alt"):
                item["alt"] = image["alt"]
            normalized.append(item)
    return normalized or None


def normalize_result(provider: str, url: str, title: str = "", content: str = "", raw_content: Optional[str] = None, **extra: Any) -> Dict[str, Any]:
    result = {
        "url": url,
        "title": title or title_from_url(url),
        "content": content or "",
        "raw_content": raw_content if raw_content is not None else content or "",
        "provider": provider,
    }
    for key, value in extra.items():
        if value is not None:
            result[key] = value
    return result


def get_extract_api_key(provider: str) -> Optional[str]:
    env_map = {
        "firecrawl": "FIRECRAWL_API_KEY",
        "linkup": "LINKUP_API_KEY",
        "tavily": "TAVILY_API_KEY",
        "exa": "EXA_API_KEY",
        "you": "YOU_API_KEY",
    }
    return os.environ.get(env_map[provider])


def extract_firecrawl(urls: List[str], api_key: str, output_format: str = "markdown", include_images: bool = False, include_raw_html: bool = False, render_js: bool = False, api_url: str = "https://api.firecrawl.dev/v2/scrape", timeout: int = 60) -> Dict[str, Any]:
    formats = ["html"] if output_format == "html" else ["markdown"]
    if include_raw_html and "html" not in formats:
        formats.append("html")
    results = []
    for url in urls:
        try:
            body: Dict[str, Any] = {"url": url, "formats": formats}
            if render_js:
                body["waitFor"] = 1000
            data = request_json(api_url, {
                "method": "POST",
                "headers": {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
                "body": json.dumps(body),
            }, timeout)
            if data.get("success") is False:
                results.append(normalize_result("firecrawl", url, error=str(data.get("error") or data.get("warning") or "Firecrawl scrape failed")))
                continue
            payload = data.get("data") if isinstance(data.get("data"), dict) else data
            metadata = payload.get("metadata") if isinstance(payload.get("metadata"), dict) else {}
            final_url = metadata.get("sourceURL") or metadata.get("url") or url
            title = metadata.get("title") or ""
            markdown = str(payload.get("markdown") or "")
            html = str(payload.get("html") or payload.get("rawHtml") or "")
            content = html if output_format == "html" else (markdown or html)
            images = None
            if include_images:
                seen = set()
                parsed = []
                og_image = metadata.get("ogImage") or metadata.get("og:image")
                if isinstance(og_image, str) and og_image and og_image not in seen:
                    parsed.append({"alt": "og:image", "url": og_image})
                    seen.add(og_image)
                import re
                for match in re.finditer(r'!\[([^\]]*)\]\(([^)]+)\)', markdown):
                    image_url = match.group(2)
                    if image_url and image_url not in seen:
                        item = {"url": image_url}
                        if match.group(1):
                            item["alt"] = match.group(1)
                        parsed.append(item)
                        seen.add(image_url)
                images = parsed or None
            results.append(normalize_result("firecrawl", final_url, title, content, content, raw_html=html or None, images=images, metadata=metadata or None))
        except Exception as exc:
            results.append(normalize_result("firecrawl", url, error=str(exc)))
    return {"provider": "firecrawl", "results": results}


def extract_linkup(urls: List[str], api_key: str, output_format: str = "markdown", include_images: bool = False, include_raw_html: bool = False, render_js: bool = False, api_url: str = "https://api.linkup.so/v1/fetch", timeout: int = 30) -> Dict[str, Any]:
    results = []
    for url in urls:
        try:
            data = request_json(api_url, {
                "method": "POST",
                "headers": {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
                "body": json.dumps({
                    "url": url,
                    "extractImages": include_images,
                    "includeRawHtml": include_raw_html or output_format == "html",
                    "renderJs": render_js,
                }),
            }, timeout)
            if data.get("error"):
                results.append(normalize_result("linkup", url, error=str(data["error"])))
                continue
            markdown = str(data.get("markdown") or "")
            raw_html = str(data.get("rawHtml") or data.get("raw_html") or "")
            content = raw_html if output_format == "html" else (markdown or raw_html)
            results.append(normalize_result("linkup", url, content=content, raw_content=content, raw_html=raw_html or None, images=normalize_images(data.get("images")) if include_images else None, metadata=data.get("metadata") if isinstance(data.get("metadata"), dict) else None))
        except Exception as exc:
            results.append(normalize_result("linkup", url, error=str(exc)))
    return {"provider": "linkup", "results": results}


def extract_tavily(urls: List[str], api_key: str, output_format: str = "markdown", include_images: bool = False, include_raw_html: bool = False, render_js: bool = False, api_url: str = "https://api.tavily.com/extract", timeout: int = 30) -> Dict[str, Any]:
    del output_format, include_raw_html, render_js
    data = request_json(api_url, {
        "method": "POST",
        "headers": {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
        "body": json.dumps({"urls": urls, "include_images": include_images}),
    }, timeout)
    results = []
    for item in data.get("results") or []:
        content = str(item.get("raw_content") or item.get("content") or "")
        results.append(normalize_result("tavily", str(item.get("url") or ""), str(item.get("title") or ""), content, content, images=normalize_images(item.get("images")) if include_images else None, metadata=item.get("metadata") if isinstance(item.get("metadata"), dict) else None))
    for failed in data.get("failed_results") or []:
        results.append(normalize_result("tavily", str(failed.get("url") or ""), error=str(failed.get("error") or "Tavily extract failed")))
    return {"provider": "tavily", "results": results}


def extract_exa(urls: List[str], api_key: str, output_format: str = "markdown", include_images: bool = False, include_raw_html: bool = False, render_js: bool = False, api_url: str = "https://api.exa.ai/contents", timeout: int = 30) -> Dict[str, Any]:
    del output_format, include_raw_html, render_js
    data = request_json(api_url, {
        "method": "POST",
        "headers": {"x-api-key": api_key, "Content-Type": "application/json"},
        "body": json.dumps({"urls": urls, "text": True}),
    }, timeout)
    results = []
    for item in data.get("results") or []:
        url = str(item.get("url") or item.get("id") or "")
        content = str(item.get("text") or item.get("summary") or "")
        metadata = {}
        for src, dest in [("summary", "summary"), ("highlights", "highlights"), ("publishedDate", "published_date"), ("author", "author"), ("favicon", "favicon")]:
            if item.get(src) is not None:
                metadata[dest] = item.get(src)
        images = [{"alt": "image", "url": str(item.get("image"))}] if include_images and item.get("image") else None
        results.append(normalize_result("exa", url, str(item.get("title") or ""), content, content, images=images, metadata=metadata or None))
    return {"provider": "exa", "results": results}


def extract_you(urls: List[str], api_key: str, output_format: str = "markdown", include_images: bool = False, include_raw_html: bool = False, render_js: bool = False, api_url: str = "https://ydc-index.io/v1/contents", timeout: int = 30) -> Dict[str, Any]:
    del include_images, render_js
    formats = ["html"] if output_format == "html" else ["markdown"]
    if include_raw_html and "html" not in formats:
        formats.append("html")
    if "metadata" not in formats:
        formats.append("metadata")
    data = request_json(api_url, {
        "method": "POST",
        "headers": {"X-API-Key": api_key, "Content-Type": "application/json"},
        "body": json.dumps({"urls": urls, "formats": formats, "crawl_timeout": max(1, min(timeout, 60))}),
    }, timeout)
    raw_items = data if isinstance(data, list) else data.get("results") or data.get("data") or []
    results = []
    for item in raw_items:
        url = str(item.get("url") or "")
        markdown = str(item.get("markdown") or "")
        html = str(item.get("html") or "")
        content = html if output_format == "html" else (markdown or html)
        results.append(normalize_result("you", url, str(item.get("title") or ""), content, content, raw_html=html or None, metadata=item.get("metadata") if isinstance(item.get("metadata"), dict) else None))
    return {"provider": "you", "results": results}


def extract_plus(urls: List[str], provider: str = "auto", output_format: str = "markdown", include_images: bool = False, include_raw_html: bool = False, render_js: bool = False) -> Dict[str, Any]:
    requested_provider = provider or "auto"
    if not urls:
        return {"provider": requested_provider, "results": [], "error": "No URLs provided", "routing": {"requested_provider": requested_provider}}
    cleaned_urls = [url.strip() for url in urls if isinstance(url, str)]
    invalid_urls = [url for url in cleaned_urls if not url.startswith(("http://", "https://"))]
    if invalid_urls:
        return {"provider": requested_provider, "results": [], "error": f"Invalid URL(s) — must start with http:// or https://: {json.dumps(invalid_urls)}", "routing": {"requested_provider": requested_provider}}
    providers = EXTRACT_PROVIDER_PRIORITY if requested_provider == "auto" else [requested_provider] + [p for p in EXTRACT_PROVIDER_PRIORITY if p != requested_provider]
    errors = []
    for current_provider in providers:
        if current_provider not in EXTRACT_PROVIDER_PRIORITY:
            errors.append({"provider": current_provider, "error": f"Provider {current_provider} does not support extraction"})
            continue
        credential = get_extract_api_key(current_provider)
        if not credential:
            errors.append({"provider": current_provider, "error": "missing_api_key"})
            continue
        try:
            if current_provider == "firecrawl":
                result = extract_firecrawl(cleaned_urls, credential, output_format, include_images, include_raw_html, render_js)
            elif current_provider == "linkup":
                result = extract_linkup(cleaned_urls, credential, output_format, include_images, include_raw_html, render_js)
            elif current_provider == "tavily":
                result = extract_tavily(cleaned_urls, credential, output_format, include_images, include_raw_html, render_js)
            elif current_provider == "exa":
                result = extract_exa(cleaned_urls, credential, output_format, include_images, include_raw_html, render_js)
            else:
                result = extract_you(cleaned_urls, credential, output_format, include_images, include_raw_html, render_js)
            result_list = result.get("results") or []
            if result_list and all(item.get("error") for item in result_list):
                errors.append({"provider": current_provider, "error": "all_urls_failed", "details": [item.get("error") for item in result_list]})
                continue
            result["routing"] = {
                "provider": current_provider,
                "requested_provider": requested_provider,
                "fallback_used": bool(errors),
                "fallback_errors": errors,
            }
            return result
        except Exception as exc:
            errors.append({"provider": current_provider, "error": str(exc)})
    return {
        "provider": requested_provider,
        "results": [],
        "error": "All extraction providers failed",
        "fallback_errors": errors,
        "routing": {"requested_provider": requested_provider, "fallback_used": bool(errors), "fallback_errors": errors},
    }


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Extract URL content with automatic provider fallback")
    parser.add_argument("--url", dest="urls", action="append", help="URL to extract (repeatable)")
    parser.add_argument("--provider", default="auto", choices=["auto"] + EXTRACT_PROVIDER_PRIORITY)
    parser.add_argument("--format", default="markdown", choices=["markdown", "html"])
    parser.add_argument("--include-images", action="store_true")
    parser.add_argument("--include-raw-html", action="store_true")
    parser.add_argument("--render-js", action="store_true")
    parser.add_argument("--compact", action="store_true", help="Print compact JSON")
    return parser.parse_args()


def main() -> None:
    args = parse_args()
    result = extract_plus(args.urls or [], args.provider, args.format, args.include_images, args.include_raw_html, args.render_js)
    if args.compact:
        print(json.dumps(result, ensure_ascii=False))
    else:
        print(json.dumps(result, ensure_ascii=False, indent=2))
    if result.get("error"):
        sys.exit(1)


if __name__ == "__main__":
    main()


path: scripts/setup.py
content: #!/usr/bin/env python3
"""
Web Search Plus - Interactive Setup Wizard
==========================================

Runs on first use (when no config.json exists) to configure providers and API keys.
Creates config.json with your settings. API keys are stored locally only.

Usage:
    python3 scripts/setup.py          # Interactive setup
    python3 scripts/setup.py --reset  # Reset and reconfigure
"""

import json
import os
import sys
from pathlib import Path

# ANSI colors for terminal output
class Colors:
    HEADER = '\033[95m'
    BLUE = '\033[94m'
    CYAN = '\033[96m'
    GREEN = '\033[92m'
    YELLOW = '\033[93m'
    RED = '\033[91m'
    BOLD = '\033[1m'
    DIM = '\033[2m'
    RESET = '\033[0m'

def color(text: str, c: str) -> str:
    """Wrap text in color codes."""
    return f"{c}{text}{Colors.RESET}"

def print_header():
    """Print the setup wizard header."""
    print()
    print(color("╔════════════════════════════════════════════════════════════╗", Colors.CYAN))
    print(color("║          🔍 Web Search Plus - Setup Wizard                 ║", Colors.CYAN))
    print(color("╚════════════════════════════════════════════════════════════╝", Colors.CYAN))
    print()
    print(color("This wizard will help you configure your search providers.", Colors.DIM))
    print(color("API keys are stored locally in config.json (gitignored).", Colors.DIM))
    print()

def print_provider_info():
    """Print information about each provider."""
    print(color("📚 Available Providers:", Colors.BOLD))
    print()

    providers = [
        {
            "name": "Serper",
            "emoji": "🔎",
            "best_for": "Google-style general web, shopping, local, news",
            "free_tier": "2,500 queries/month",
            "signup": "https://serper.dev",
            "strengths": ["Shopping and local intent", "Knowledge Graph", "Broad fallback coverage"]
        },
        {
            "name": "Brave",
            "emoji": "🦁",
            "best_for": "Independent general web index, broad factual queries",
            "free_tier": "$5/mo free credits",
            "signup": "https://brave.com/search/api/",
            "strengths": ["Independent index", "Good generic-web fallback", "Pairs well with Serper tie-breaking"]
        },
        {
            "name": "Tavily",
            "emoji": "📖",
            "best_for": "Research, explanations, in-depth analysis",
            "free_tier": "1,000 queries/month",
            "signup": "https://tavily.com",
            "strengths": ["Research intent", "Full page content", "Academic-friendly"]
        },
        {
            "name": "Querit",
            "emoji": "🌍",
            "best_for": "Multilingual, international, recency-aware AI search",
            "free_tier": "Varies",
            "signup": "https://querit.ai",
            "strengths": ["Cross-language queries", "International updates", "AI-ready metadata"]
        },
        {
            "name": "Linkup",
            "emoji": "🔗",
            "best_for": "Source-grounded search, citations, evidence-first retrieval",
            "free_tier": "€5 free monthly credits",
            "signup": "https://linkup.so",
            "strengths": ["Citation-heavy queries", "Grounding", "Good companion to extraction"]
        },
        {
            "name": "Exa",
            "emoji": "🧠",
            "best_for": "Semantic search, discovery, deep and deep-reasoning modes",
            "free_tier": "1,000 queries/month",
            "signup": "https://exa.ai",
            "strengths": ["Semantic discovery", "Similar pages", "Deep synthesis"]
        },
        {
            "name": "Firecrawl",
            "emoji": "🔥",
            "best_for": "Search with scrape-ready content and extraction-friendly metadata",
            "free_tier": "Free plan credits",
            "signup": "https://www.firecrawl.dev/app/api-keys",
            "strengths": ["Search + scrape metadata", "Recency filters", "Great extraction fallback"]
        },
        {
            "name": "Perplexity via Kilo",
            "emoji": "⚡",
            "best_for": "Direct answers with citations",
            "free_tier": "Depends on Kilo plan",
            "signup": "https://kilo.ai",
            "strengths": ["Answer-first output", "Current status questions", "Citation support"]
        },
        {
            "name": "You.com",
            "emoji": "🤖",
            "best_for": "RAG applications, real-time info, LLM-ready snippets",
            "free_tier": "Limited free tier",
            "signup": "https://api.you.com",
            "strengths": ["LLM-ready snippets", "Combined web + news", "Live content API"]
        },
        {
            "name": "SearXNG",
            "emoji": "🔒",
            "best_for": "Privacy-first search, multi-source aggregation, $0 API cost",
            "free_tier": "FREE (self-hosted)",
            "signup": "https://docs.searxng.org/admin/installation.html",
            "strengths": ["Privacy-preserving", "70+ search engines", "Self-hosted control"]
        }
    ]

    for p in providers:
        print(f"  {p['emoji']} {color(p['name'], Colors.BOLD)}")
        print(f"     Best for: {color(p['best_for'], Colors.GREEN)}")
        print(f"     Free tier: {p['free_tier']}")
        print(f"     Sign up: {color(p['signup'], Colors.BLUE)}")
        print()

def ask_yes_no(prompt: str, default: bool = True) -> bool:
    """Ask a yes/no question."""
    suffix = "[Y/n]" if default else "[y/N]"
    while True:
        response = input(f"{prompt} {color(suffix, Colors.DIM)}: ").strip().lower()
        if response == "":
            return default
        if response in ("y", "yes"):
            return True
        if response in ("n", "no"):
            return False
        print(color("  Please enter 'y' or 'n'", Colors.YELLOW))

def ask_choice(prompt: str, options: list, default: str = None) -> str:
    """Ask user to choose from a list of options."""
    print(f"\n{prompt}")
    for i, opt in enumerate(options, 1):
        marker = color("→", Colors.GREEN) if opt == default else " "
        print(f"  {marker} {i}. {opt}")

    while True:
        hint = f" [default: {default}]" if default else ""
        response = input(f"Enter number (1-{len(options)}){color(hint, Colors.DIM)}: ").strip()

        if response == "" and default:
            return default

        try:
            idx = int(response)
            if 1 <= idx <= len(options):
                return options[idx - 1]
        except ValueError:
            pass

        print(color(f"  Please enter a number between 1 and {len(options)}", Colors.YELLOW))

def ask_api_key(provider: str, signup_url: str) -> str:
    """Ask for an API key with validation."""
    print()
    print(f"  {color(f'Get your {provider} API key:', Colors.DIM)} {color(signup_url, Colors.BLUE)}")

    while True:
        key = input(f"  Enter your {provider} API key: ").strip()

        if not key:
            print(color("    ⚠️  No key entered. This provider will be disabled.", Colors.YELLOW))
            return None

        # Basic validation
        if len(key) < 10:
            print(color("    ⚠️  Key seems too short. Please check and try again.", Colors.YELLOW))
            continue

        # Mask key for confirmation
        masked = key[:4] + "..." + key[-4:] if len(key) > 12 else key[:2] + "..."
        print(color(f"    ✓ Key saved: {masked}", Colors.GREEN))
        return key


def ask_searxng_instance(docs_url: str) -> str:
    """Ask for SearXNG instance URL with connection test."""
    print()
    print(f"  {color('SearXNG is self-hosted. You need your own instance.', Colors.DIM)}")
    print(f"  {color('Setup guide:', Colors.DIM)} {color(docs_url, Colors.BLUE)}")
    print()
    print(f"  {color('Example URLs:', Colors.DIM)}")
    print(f"    • http://localhost:8080 (local Docker)")
    print(f"    • https://searx.your-domain.com (self-hosted)")
    print()

    while True:
        url = input(f"  Enter your SearXNG instance URL: ").strip()

        if not url:
            print(color("    ⚠️  No URL entered. SearXNG will be disabled.", Colors.YELLOW))
            return None

        # Basic URL validation
        if not url.startswith(("http://", "https://")):
            print(color("    ⚠️  URL must start with http:// or https://", Colors.YELLOW))
            continue

        # SSRF protection: validate URL before connecting
        try:
            import ipaddress
            import socket
            from urllib.parse import urlparse as _urlparse
            _parsed = _urlparse(url)
            _hostname = _parsed.hostname or ""
            _blocked = {"169.254.169.254", "metadata.google.internal", "metadata.internal"}
            if _hostname in _blocked:
                print(color(f"    ❌ Blocked: {_hostname} is a cloud metadata endpoint.", Colors.RED))
                continue
            if not os.environ.get("SEARXNG_ALLOW_PRIVATE", "").strip() == "1":
                _resolved = socket.getaddrinfo(_hostname, _parsed.port or 80, proto=socket.IPPROTO_TCP)
                for _fam, _t, _p, _cn, _sa in _resolved:
                    _ip = ipaddress.ip_address(_sa[0])
                    if _ip.is_loopback or _ip.is_private or _ip.is_link_local or _ip.is_reserved:
                        print(color(f"    ❌ Blocked: {_hostname} resolves to private IP {_ip}.", Colors.RED))
                        print(color(f"       Set SEARXNG_ALLOW_PRIVATE=1 if intentional.", Colors.DIM))
                        raise ValueError("private_ip")
        except ValueError as _ve:
            if str(_ve) == "private_ip":
                continue
            raise
        except socket.gaierror:
            print(color(f"    ❌ Cannot resolve hostname: {_hostname}", Colors.RED))
            continue

        # Test connection
        print(color(f"    Testing connection to {url}...", Colors.DIM))
        try:
            import urllib.request
            import urllib.error

            test_url = f"{url.rstrip('/')}/search?q=test&format=json"
            req = urllib.request.Request(
                test_url,
                headers={"User-Agent": "ClawdBot-WebSearchPlus/2.5", "Accept": "application/json"}
            )

            with urllib.request.urlopen(req, timeout=10) as response:
                data = response.read().decode("utf-8")
                import json
                result = json.loads(data)

                # Check if it looks like SearXNG JSON response
                if "results" in result or "query" in result:
                    print(color(f"    ✓ Connection successful! SearXNG instance is working.", Colors.GREEN))
                    return url.rstrip("/")
                else:
                    print(color(f"    ⚠️  Connected but response doesn't look like SearXNG JSON.", Colors.YELLOW))
                    if ask_yes_no("    Use this URL anyway?", default=False):
                        return url.rstrip("/")

        except urllib.error.HTTPError as e:
            if e.code == 403:
                print(color(f"    ⚠️  JSON API is disabled (403 Forbidden).", Colors.YELLOW))
                print(color(f"       Enable JSON in settings.yml: search.formats: [html, json]", Colors.DIM))
            else:
                print(color(f"    ⚠️  HTTP error: {e.code} {e.reason}", Colors.YELLOW))

            if ask_yes_no("    Try a different URL?", default=True):
                continue
            return None

        except urllib.error.URLError as e:
            print(color(f"    ⚠️  Cannot reach instance: {e.reason}", Colors.YELLOW))
            if ask_yes_no("    Try a different URL?", default=True):
                continue
            return None

        except Exception as e:
            print(color(f"    ⚠️  Error: {e}", Colors.YELLOW))
            if ask_yes_no("    Try a different URL?", default=True):
                continue
            return None

def ask_result_count() -> int:
    """Ask for default result count."""
    options = ["3 (fast, minimal)", "5 (balanced - recommended)", "10 (comprehensive)"]
    choice = ask_choice("Default number of results per search?", options, "5 (balanced - recommended)")

    if "3" in choice:
        return 3
    elif "10" in choice:
        return 10
    return 5

def run_setup(skill_dir: Path, force_reset: bool = False):
    """Run the interactive setup wizard."""
    config_path = skill_dir / "config.json"
    example_path = skill_dir / "config.example.json"

    # Check if config already exists
    if config_path.exists() and not force_reset:
        print(color("✓ config.json already exists!", Colors.GREEN))
        print()
        if not ask_yes_no("Do you want to reconfigure?", default=False):
            print(color("Setup cancelled. Your existing config is unchanged.", Colors.DIM))
            return False
        print()

    print_header()
    print_provider_info()

    # Load example config as base
    if example_path.exists():
        with open(example_path) as f:
            config = json.load(f)
    else:
        config = {
            "defaults": {"provider": "serper", "max_results": 5},
            "auto_routing": {"enabled": True, "fallback_provider": "serper"},
            "serper": {},
            "brave": {},
            "tavily": {},
            "querit": {},
            "linkup": {},
            "exa": {},
            "firecrawl": {},
            "perplexity": {},
            "you": {},
            "searxng": {},
            "serpbase": {}
        }

    # Remove any existing API keys from example
    for provider in ["serper", "brave", "tavily", "querit", "linkup", "exa", "firecrawl", "perplexity", "you", "serpbase"]:
        if provider in config:
            config[provider].pop("api_key", None)

    enabled_providers = []

    # ===== Question 1: Which providers to enable =====
    print(color("─" * 60, Colors.DIM))
    print(color("\n📋 Step 1: Choose Your Providers\n", Colors.BOLD))
    print("Select which search providers you want to enable.")
    print(color("(You need at least one API key to use this skill)", Colors.DIM))
    print()

    providers_info = {
        "serper": ("Serper", "https://serper.dev", "Google-style results, shopping, local"),
        "brave": ("Brave", "https://brave.com/search/api/", "Independent general web search"),
        "tavily": ("Tavily", "https://tavily.com", "Research, explanations, analysis"),
        "querit": ("Querit", "https://querit.ai", "Multilingual and international AI search"),
        "linkup": ("Linkup", "https://linkup.so", "Citation and source-grounded search"),
        "exa": ("Exa", "https://exa.ai", "Semantic search, discovery, deep synthesis"),
        "firecrawl": ("Firecrawl", "https://www.firecrawl.dev/app/api-keys", "Search with scrape-ready metadata"),
        "perplexity": ("Perplexity via Kilo", "https://kilo.ai", "Direct answers with citations"),
        "you": ("You.com", "https://api.you.com", "RAG applications, real-time info"),
        "searxng": ("SearXNG", "https://docs.searxng.org/admin/installation.html", "Privacy-first, self-hosted, $0 cost"),
        "serpbase": ("SerpBase", "https://serpbase.dev", "Low-cost Google SERP, prepaid credits (explicit/fallback-only)")
    }

    for provider, (name, url, desc) in providers_info.items():
        print(f"  {color(name, Colors.BOLD)}: {desc}")

        # Special handling for SearXNG
        if provider == "searxng":
            print(color("    Note: SearXNG requires a self-hosted instance (no API key needed)", Colors.DIM))
            if ask_yes_no(f"    Do you have a SearXNG instance?", default=False):
                instance_url = ask_searxng_instance(url)
                if instance_url:
                    if "searxng" not in config:
                        config["searxng"] = {}
                    config["searxng"]["instance_url"] = instance_url
                    enabled_providers.append(provider)
                else:
                    print(color(f"    → {name} disabled (no instance URL)", Colors.DIM))
            else:
                print(color(f"    → {name} skipped (no instance)", Colors.DIM))
        else:
            if ask_yes_no(f"    Enable {name}?", default=True):
                # ===== Question 2: API key for each enabled provider =====
                [REDACTED_SECRET], url)
                if [REDACTED_SECRET]"api_key"] = api_key
                    enabled_providers.append(provider)
                else:
                    print(color(f"    → {name} disabled (no API key)", Colors.DIM))
            else:
                print(color(f"    → {name} disabled", Colors.DIM))
        print()

    if not enabled_providers:
        print()
        print(color("⚠️  No providers enabled!", Colors.RED))
        print("You need at least one API key to use web-search-plus.")
        print("Run this setup again when you have an API key.")
        return False

    # ===== Question 3: Default provider =====
    print(color("─" * 60, Colors.DIM))
    print(color("\n⚙️  Step 2: Default Settings\n", Colors.BOLD))

    if len(enabled_providers) > 1:
        default_provider = ask_choice(
            "Which provider should be the default for general queries?",
            enabled_providers,
            enabled_providers[0]
        )
    else:
        default_provider = enabled_providers[0]
        print(f"Default provider: {color(default_provider, Colors.GREEN)} (only one enabled)")

    config["defaults"]["provider"] = default_provider
    config["auto_routing"]["fallback_provider"] = default_provider

    # ===== Question 4: Auto-routing =====
    print()
    print(color("Auto-routing", Colors.BOLD) + " automatically picks the best provider for each query:")
    print(color("  • 'weather in Vienna today' → Brave or Serper (generic current web)", Colors.DIM))
    print(color("  • 'find credible sources for X' → Linkup (source-grounded intent)", Colors.DIM))
    print(color("  • 'deep research on X' → Exa deep/deep-reasoning or Tavily", Colors.DIM))
    print()

    auto_routing = ask_yes_no("Enable auto-routing?", default=True)
    config["auto_routing"]["enabled"] = auto_routing

    if not auto_routing:
        print(color(f"  → All queries will use {default_provider}", Colors.DIM))

    # ===== Question 5: Result count =====
    print()
    max_results = ask_result_count()
    config["defaults"]["max_results"] = max_results

    # Set disabled providers (serpbase is excluded from default auto-routing — opt-in only)
    all_providers = ["serper", "brave", "tavily", "querit", "linkup", "exa", "firecrawl", "perplexity", "you", "searxng", "serpbase"]
    disabled = [p for p in all_providers if p not in enabled_providers]
    config["auto_routing"]["disabled_providers"] = disabled

    # ===== Save config =====
    print()
    print(color("─" * 60, Colors.DIM))
    print(color("\n💾 Saving Configuration\n", Colors.BOLD))

    with open(config_path, 'w') as f:
        json.dump(config, f, indent=2)

    print(color(f"✓ Configuration saved to: {config_path}", Colors.GREEN))
    print()

    # ===== Summary =====
    print(color("📋 Configuration Summary:", Colors.BOLD))
    print(f"   Enabled providers: {', '.join(enabled_providers)}")
    print(f"   Default provider: {default_provider}")
    print(f"   Auto-routing: {'enabled' if auto_routing else 'disabled'}")
    print(f"   Results per search: {max_results}")
    print()

    # ===== Test suggestion =====
    print(color("🚀 Ready to search! Try:", Colors.BOLD))
    print(color(f"   python3 scripts/search.py -q \"your query here\"", Colors.CYAN))
    print()

    return True

def check_first_run(skill_dir: Path) -> bool:
    """Check if this is the first run (no config.json)."""
    config_path = skill_dir / "config.json"
    return not config_path.exists()

def main():
    # Determine skill directory
    script_path = Path(__file__).resolve()
    skill_dir = script_path.parent.parent

    # Check for --reset flag
    force_reset = "--reset" in sys.argv

    # Check for --check flag (just check if setup needed)
    if "--check" in sys.argv:
        if check_first_run(skill_dir):
            print("Setup required: config.json not found")
            sys.exit(1)
        else:
            print("Setup complete: config.json exists")
            sys.exit(0)

    # Run setup
    success = run_setup(skill_dir, force_reset)
    sys.exit(0 if success else 1)

if __name__ == "__main__":
    main()


path: test-auto-routing.sh
content: #!/bin/bash

# Test Auto-Routing Feature
# Tests various query types to verify routing works correctly

# Load from environment or .env file
if [ -f .env ]; then
  source .env
fi

# Check required keys
if [ -z "$SERPER_API_KEY" ]; then
  echo "Error: SERPER_API_KEY not set. Copy .env.example to .env and add your keys."
  exit 1
fi

echo "Testing auto-routing..."
python3 scripts/search.py -q "buy iPhone 15 price" --auto
python3 scripts/search.py -q "how does quantum computing work" --auto  
python3 scripts/search.py -q "companies like Stripe" --auto


path: tests/test_routing_and_extract.py
content: import os
import sys
import unittest
from pathlib import Path
from unittest import mock

SKILL_DIR = Path(__file__).resolve().parents[1]
if str(SKILL_DIR) not in sys.path:
    sys.path.insert(0, str(SKILL_DIR))

from scripts import extract, search


class RoutingParityTests(unittest.TestCase):
    def make_analyzer(self):
        config = {
            "auto_routing": {
                "fallback_provider": "serper",
                "provider_priority": [
                    "tavily",
                    "linkup",
                    "querit",
                    "exa",
                    "firecrawl",
                    "perplexity",
                    "brave",
                    "serper",
                    "you",
                    "searxng",
                ],
                "disabled_providers": ["searxng"],
            },
            "serper": {"api_key": "x"},
            "brave": {"api_key": "x"},
            "tavily": {"api_key": "x"},
            "querit": {"api_key": "x"},
            "linkup": {"api_key": "x"},
            "exa": {"api_key": "x"},
            "firecrawl": {"api_key": "x"},
            "perplexity": {"api_key": "x"},
            "you": {"api_key": "x"},
            "searxng": {},
        }
        return search.QueryAnalyzer(config)

    def test_linkup_routing_for_source_grounded_query(self):
        routed = self.make_analyzer().route("find credible sources and citations for AI tutoring outcomes")
        self.assertEqual(routed["provider"], "linkup")

    def test_multilingual_recency_query_scores_querit_signal(self):
        routed = self.make_analyzer().route("latest AI policy updates in Germany")
        self.assertGreater(routed["scores"]["querit"], 0)
        self.assertIn(routed["provider"], {"serper", "brave", "perplexity", "querit", "tavily"})

    def test_generic_current_web_query_uses_brave_or_serper(self):
        routed = self.make_analyzer().route("weather in Vienna today")
        self.assertIn(routed["provider"], {"brave", "serper", "perplexity"})

    def test_tie_breaker_is_deterministic(self):
        winners = ["brave", "serper"]
        first = search._choose_tie_winner("weather in Vienna today", winners, winners)
        second = search._choose_tie_winner("weather in Vienna today", winners, winners)
        self.assertEqual(first, second)
        self.assertIn(first, winners)


class ExtractTests(unittest.TestCase):
    def test_invalid_url_rejected(self):
        result = extract.extract_plus(["example.com"], provider="auto")
        self.assertIn("Invalid URL", result["error"])

    def test_missing_keys_reported(self):
        with mock.patch.dict(os.environ, {}, clear=True):
            result = extract.extract_plus(["https://example.com"], provider="auto")
        self.assertEqual(result["error"], "All extraction providers failed")
        self.assertEqual(result["fallback_errors"][0]["provider"], "firecrawl")
        self.assertEqual(result["fallback_errors"][0]["error"], "missing_api_key")

    def test_auto_fallback_uses_next_provider_after_failure(self):
        with mock.patch.dict(os.environ, {"FIRECRAWL_API_KEY": "fire", "LINKUP_API_KEY": "link"}, clear=True):
            with mock.patch.object(extract, "extract_firecrawl", side_effect=RuntimeError("boom")):
                with mock.patch.object(extract, "extract_linkup", return_value={
                    "provider": "linkup",
                    "results": [{"url": "https://example.com", "title": "Example", "content": "ok", "raw_content": "ok", "provider": "linkup"}],
                }):
                    result = extract.extract_plus(["https://example.com"], provider="auto")
        self.assertEqual(result["provider"], "linkup")
        self.assertTrue(result["routing"]["fallback_used"])
        self.assertEqual(result["routing"]["fallback_errors"][0]["provider"], "firecrawl")


if __name__ == "__main__":
    unittest.main()


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal:  | SkillSpector: suspicious (CRITICAL)