Skill: sky-lv/openclaw-ask-search (version 1.0.0)

--- name: openclaw-ask-search description: "Web search via self-hosted SearxNG. Aggregates Google, Bing, DuckDuckGo, Brave. Returns title/url/snippet. Zero API key required." --- # ask-search Web search powered by [SearxNG](https://github.com/searxng/searxng). Aggregates multiple search engines, zero API key, full privacy. ## Usage ```bash ask-search "your query" # top 10 results ask-search "query" --num 5 # limit results ask-search "AI news" --categories news # news only ask-search "query" --lang zh-CN # Chinese results ask-search "query" --urls-only # URL list (pipe to web_fetch) ask-search "query" --json # raw JSON ``` ## Agent Workflow 1. Run `ask-search "topic"` to get candidates 2. Check snippet — if enough, answer directly 3. If snippet truncated, use `web_fetch` on the URL for full content ## Parameters | Flag | Short | Description | |------|-------|-------------| | `--num N` | `-n` | Max results (default 10) | | `--engines` | `-e` | google,bing,duckduckgo,brave | | `--lang` | `-l` | zh-CN, en, ja, ko | | `--categories` | `-c` | general,news,images,science | | `--json` | `-j` | Raw JSON output | | `--urls-only` | `-u` | URLs only | ## Setup Requires SearxNG running locally. Set `SEARXNG_URL` if not on default port 8080.

path: install.sh
content: #!/bin/bash
# ask-search install script
# Installs ask-search as a global command + optionally sets up SearxNG

set -e

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
INSTALL_BIN="${1:-/usr/local/bin}"

echo "=== ask-search installer ==="

# Check dependencies
if ! command -v python3 &>/dev/null; then
    echo "ERROR: python3 is required"
    exit 1
fi
if ! command -v curl &>/dev/null; then
    echo "ERROR: curl is required"
    exit 1
fi

# Install global command
echo "Installing ask-search to $INSTALL_BIN ..."

# Create wrapper with hardcoded path (single-quotes prevent $@ expansion in heredoc)
WRAPPER=$(cat << 'WRAPPEREOF'
#!/bin/bash
exec python3 "COREPATH" "$@"
WRAPPEREOF
)

# Replace placeholder with actual path
WRAPPER="${WRAPPER/COREPATH/$SCRIPT_DIR/scripts/core.py}"
TMPFILE=$(mktemp)
echo "$WRAPPER" > "$TMPFILE"
install -m 755 "$TMPFILE" "$INSTALL_BIN/ask-search"
rm -f "$TMPFILE"
echo "✓ ask-search installed to $INSTALL_BIN/ask-search"

# Test
echo ""
echo "Testing connection to SearxNG at ${SEARXNG_URL:-http://localhost:8080} ..."
if python3 "$SCRIPT_DIR/scripts/core.py" "test" --num 1 &>/dev/null; then
    echo "✓ SearxNG reachable — ask-search is ready!"
else
    echo "⚠ SearxNG not reachable at ${SEARXNG_URL:-http://localhost:8080}"
    echo "  Set SEARXNG_URL env var or deploy SearxNG first."
    echo "  See README for SearxNG setup instructions."
fi

echo ""
echo "Usage: ask-search \"your query\""


path: mcp/server.py
content: #!/usr/bin/env python3
"""
ask-search MCP Server — for Antigravity / Claude Code MCP integration

Install:
  pip install mcp

Add to your MCP config:
  {
    "mcpServers": {
      "ask-search": {
        "command": "python3",
        "args": ["/path/to/ask-search/mcp/server.py"],
        "env": {"SEARXNG_URL": "http://localhost:8080"}
      }
    }
  }
"""
import os, sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "scripts"))

try:
    from mcp.server.fastmcp import FastMCP
except ImportError:
    print("Error: mcp package not installed. Run: pip install mcp", file=sys.stderr)
    sys.exit(1)

from core import searxng_search, search, fmt_results

mcp = FastMCP("ask-search")

@mcp.tool()
def web_search(query: str, num_results: int = 10) -> str:
    """
    Search the web via SearxNG. Returns JSON array of results with title, url, content.
    Aggregates Google, Bing, DuckDuckGo, Brave and more.

    Args:
        query: Search query string
        num_results: Number of results to return (default 10, max 20)
    """
    return searxng_search(query, min(num_results, 20))

@mcp.tool()
def web_search_news(query: str, num_results: int = 10) -> str:
    """
    Search recent news via SearxNG news category.

    Args:
        query: News search query
        num_results: Number of results (default 10)
    """
    try:
        results = search(query, num_results, categories="news")
        import json
        return json.dumps({"query": query, "category": "news", "results": results}, ensure_ascii=False, indent=2)
    except Exception as e:
        import json
        return json.dumps({"error": str(e)})

if __name__ == "__main__":
    mcp.run()


path: README.md
content: <div align="center">

# 🔍 ask-search

**Self-hosted web search for AI agents — zero API key, full privacy**

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![OpenClaw](https://img.shields.io/badge/OpenClaw-Skill-blue.svg)](https://github.com/openclaw/openclaw)
[![Python](https://img.shields.io/badge/Python-3.8%2B-blue.svg)](https://python.org)

*Works in OpenClaw · Claude Code · Antigravity · Any CLI*

</div>

---

## 😡 The Problem

Your AI agent wants to search the web, but:
- Brave Search API: $3/1000 queries, rate limited
- Google Custom Search: $5/1000, daily caps
- Bing API: paid, complex setup
- Built-in web search: sends queries to third-party servers

**You just want your local agent to search Google without paying or leaking queries.**

## ✅ The Solution

`ask-search` wraps [SearxNG](https://github.com/searxng/searxng) — a self-hosted meta search engine that aggregates Google, Bing, DuckDuckGo, Brave and 70+ more sources. One command, all results, zero cost.

```bash
ask-search "Claude Code vs Cursor 2026"
```

```
[1] Claude Code Is Eating Cursor's Lunch
    https://techcrunch.com/2026/...
    After Anthropic launched Claude Code with agent mode...
    [google,brave]

[2] Why I switched from Cursor to Claude Code
    https://reddit.com/r/LocalLLaMA/...
    ...
```

## 📊 Compatibility

| Environment | Integration | Status |
|-------------|-------------|--------|
| **OpenClaw** | CLI Skill (`SKILL.md`) | ✅ |
| **Claude Code** | CLI command | ✅ |
| **Antigravity** | MCP Server | ✅ |
| Any shell | `ask-search` CLI | ✅ |

## 🚀 Quick Start

### 30-second version (if SearxNG already running)

```bash
git clone https://github.com/ythx-101/ask-search
cd ask-search
bash install.sh
ask-search "hello world"
```

### Full setup with Docker Compose (Recommended)

```bash
# Clone and navigate to the project
git clone https://github.com/ythx-101/ask-search
cd ask-search

# Generate a secure secret key
python3 -c "import secrets; print('SEARXNG_SECRET=' + secrets.token_hex(32))"

# Copy .env.example to .env and set your secret key
cp searxng/.env.example searxng/.env
# Edit .env with your generated secret

# Start SearxNG
cd searxng && docker-compose up -d

# Configure ask-search with your secret
export SEARXNG_SECRET="your-generated-secret"
ask-search "hello world"
```

### Full setup (SearxNG + ask-search)

**Step 1: Deploy SearxNG**

```bash
# Docker (recommended)
docker run -d --name searxng \
  -p 127.0.0.1:8080:8080 \
  -e SEARXNG_SECRET_KEY=your-secret-key \
  searxng/searxng

# Or Docker Compose — see searxng/docker-compose.yml in this repo
```

**Step 2: Enable JSON output**

Edit SearxNG `settings.yml`:
```yaml
search:
  formats:
    - html
    - json
```

**Step 3: Install ask-search**

```bash
bash install.sh
```

**Step 4: Use it**

```bash
ask-search "your query"
```

## 🔧 Usage

```bash
ask-search "query"                    # top 10 results
ask-search "query" --num 5            # limit results
ask-search "AI news" --categories news # news only
ask-search "query" --lang zh-CN       # language filter
ask-search "query" --urls-only        # URLs only (pipe to web_fetch)
ask-search "query" --json             # raw JSON
ask-search "query" -e google,brave    # specific engines
```

## ⚙️ Configuration

| Variable | Default | Description |
|----------|---------|-------------|
| `SEARXNG_URL` | `http://localhost:8080` | SearxNG endpoint |

```bash
export SEARXNG_URL="http://localhost:8080"
ask-search "query"
```

## 🤖 MCP Integration (Antigravity / Claude Code)

```json
{
  "mcpServers": {
    "ask-search": {
      "command": "python3",
      "args": ["/path/to/ask-search/mcp/server.py"],
      "env": {
        "SEARXNG_URL": "http://localhost:8080"
      }
    }
  }
}
```

Requires: `pip install mcp`

## 🦞 OpenClaw Skill

Copy `SKILL.md` to your OpenClaw skills directory, or:

```bash
# OpenClaw skill loader
skill-add https://github.com/ythx-101/ask-search
```

Then in OpenClaw:
```
ask-search "latest news about X"
```

## 🤝 Agent Workflow

```bash
# 1. Search
ask-search "React Server Components performance 2026" --num 10

# 2. Got a promising URL? Deep-dive:
# Pass the URL to web_fetch / curl for full content

# 3. News mode
ask-search "GPT-5 release" --categories news --lang en
```

## 📁 Structure

```
ask-search/
├── scripts/
│   └── core.py        # Main logic, CLI entry point
├── mcp/
│   └── server.py      # MCP server for AG/CC integration
├── install.sh         # Installer
├── SKILL.md           # OpenClaw skill descriptor
└── README.md
```

## ⚠️ SearxNG Setup Notes

- Must enable `json` in `search.formats` in `settings.yml`
- Bot detection may block requests from some IPs — add your server IP to `pass_ip` in `limiter.toml`
- Bind to `127.0.0.1` for security unless you need remote access

## 🌐 Deep-Dive Limitations & Workarounds

`ask-search` returns URLs + snippets from search engine indexes. When your agent needs the **full page content** (deep-dive via `curl` / `web_fetch`), some sites will block depending on your server's network environment:

### What works out of the box

| Site | Search (SearxNG) | Deep-dive (curl/fetch) | Why |
|------|:-:|:-:|-----|
| Most sites | ✅ | ✅ | No aggressive anti-bot |
| Reddit | ✅ | ❌ VPS IP blocked | Reddit blocks datacenter IPs |
| Zhihu (知乎) | ✅ | ❌ Login wall + fingerprint | Requires browser JS + login |
| Medium | ✅ | ⚠️ Paywall | Partial content only |

**Key insight**: Search always works because SearxNG queries search engines (Google, Brave, etc.), not the target sites directly. The search engines have already indexed the content. The problem only appears when your agent tries to fetch the full page.

### Solution 1: SOCKS proxy via residential IP

If you have a machine on a residential network (home server, laptop, etc.), create an SSH SOCKS tunnel:

```bash
# On your VPS/server:
ssh -f -N -D 127.0.0.1:1082 user@your-home-machine

# Then fetch through the proxy:
curl -x socks5h://127.0.0.1:1082 "https://reddit.com/r/example/comments/xxx.json"
```

For Reddit specifically, append `.json` to any post URL for structured data:
```bash
# Returns full post + all comments as JSON
curl -x socks5h://127.0.0.1:1082 \
  "https://www.reddit.com/r/LocalLLaMA/comments/xxxxx/post_title.json"
```

To persist the tunnel as a systemd service:
```ini
# /etc/systemd/system/socks-proxy.service
[Unit]
Description=SSH SOCKS Proxy for web scraping
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/ssh -N -D 127.0.0.1:1082 -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o ExitOnForwardFailure=yes user@your-home-machine
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
```

### Solution 2: Headless browser for JS-heavy sites

Sites like Zhihu require full browser rendering. Use Playwright with the SOCKS proxy:

```python
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=True,
        proxy={"server": "socks5://127.0.0.1:1082"}
    )
    page = browser.new_page()
    page.goto("https://example.com/article")
    content = page.inner_text("article")
```

> **Note**: Some sites (Zhihu, etc.) detect headless browsers even through proxies. For these, you may need a real browser session with login cookies, or delegate the fetch to an agent running on a local machine (e.g., Claude Code on a Mac).

### Solution 3: Leverage archive caches

When direct access fails, try cached versions:

```bash
# Archive.org (works surprisingly often for Reddit)
curl "https://web.archive.org/web/2026/https://reddit.com/r/example/comments/xxx"

# Google Cache (may redirect — not always reliable)
curl "https://webcache.googleusercontent.com/search?q=cache:example.com/page"
```

### Solution 4: Multi-node agent architecture

If you run agents on multiple machines (e.g., OpenClaw on VPS + Claude Code on local Mac):

```
VPS agent: ask-search "query" → gets URLs + snippets
                ↓
Local agent: web_fetch(url) → full content (residential IP, no blocks)
                ↓
VPS agent: receives full text, analyzes, responds
```

This is the most robust approach — search on your server, deep-dive from a local machine where anti-bot measures don't apply.

### TL;DR

| Problem | Fix |
|---------|-----|
| Reddit blocks your IP | SSH SOCKS proxy + `.json` API |
| Site needs JS rendering | Playwright + proxy |
| Site needs login (Zhihu) | Delegate to local agent or use logged-in browser |
| Everything blocked | Fall back to search snippets + archive caches |

## 🤝 Contributing

Issues and PRs welcome.

## Acknowledgements

Inspired by [Perplexica](https://github.com/ItzCrazyKns/Perplexica) —
the idea of using self-hosted SearxNG as a private search backend comes from there.
ask-search strips it down to a single CLI command for agent use.

## 📄 License

[MIT](LICENSE)


path: scripts/core.py
content: #!/usr/bin/env python3
"""
ask-search v1.0.0 — Cross-environment SearxNG search skill

Works in: OpenClaw (CLI) | Claude Code (CLI) | Antigravity (MCP)

Usage:
  ask-search "query"                    # top 10 results
  ask-search "query" --num 5            # limit results
  ask-search "AI news" --categories news
  ask-search "query" --lang zh-CN
  ask-search "query" --urls-only        # URL list only (pipe to web_fetch)
  ask-search "query" --json             # raw JSON output

Environment:
  SEARXNG_URL   SearxNG endpoint (default: http://localhost:8080)
"""
import sys, json, urllib.parse, argparse, os, subprocess

VERSION = "1.0.0"

def _search_url():
    base = os.environ.get("SEARXNG_URL", "http://localhost:8080")
    return base.rstrip("/") + "/search"

def search(query, num=10, engines=None, lang=None, categories=None):
    params = {"q": query, "format": "json", "pageno": 1}
    if engines:    params["engines"] = engines
    if lang:       params["language"] = lang
    if categories: params["categories"] = categories

    url = _search_url() + "?" + urllib.parse.urlencode(params)
    # Use subprocess curl to avoid urllib HTTP/1.0 compatibility issues
    result = subprocess.run(
        ["curl", "-s", "--max-time", "15", url],
        capture_output=True, text=True, timeout=20
    )
    if result.returncode != 0:
        raise RuntimeError(f"curl failed: {result.stderr[:200]}")
    data = json.loads(result.stdout)
    return data.get("results", [])[:num]

def fmt_results(results, urls_only=False):
    if urls_only:
        return "\n".join(r.get("url", "") for r in results)
    lines = []
    for i, r in enumerate(results, 1):
        title   = r.get("title", "").strip()
        url     = r.get("url", "")
        content = r.get("content", "").strip()
        engines = ",".join(r.get("engines", []))
        lines.append(f"[{i}] {title}")
        lines.append(f"    {url}")
        if content: lines.append(f"    {content[:200]}")
        if engines: lines.append(f"    [{engines}]")
        lines.append("")
    return "\n".join(lines).strip()

def searxng_search(query, num=15):
    """MCP/legacy interface — returns JSON string"""
    try:
        results = search(query, num)
        return json.dumps({"query": query, "results": results}, ensure_ascii=False, indent=2)
    except Exception as e:
        return json.dumps({"error": str(e)})

def main():
    p = argparse.ArgumentParser(description="SearxNG search (cross-environment)")
    p.add_argument("query")
    p.add_argument("--num",        "-n", type=int, default=10)
    p.add_argument("--engines",    "-e", help="google,bing,duckduckgo,brave")
    p.add_argument("--lang",       "-l", help="zh-CN, en, ja, ko")
    p.add_argument("--categories", "-c", help="general,news,images,science")
    p.add_argument("--json",       "-j", action="store_true")
    p.add_argument("--urls-only",  "-u", action="store_true")
    p.add_argument("--version",    "-V", action="version", version=f"ask-search {VERSION}")
    args = p.parse_args()

    try:
        results = search(args.query, args.num, args.engines, args.lang, args.categories)
    except Exception as e:
        print(json.dumps({"error": str(e)})); sys.exit(1)

    if not results:
        print(json.dumps({"error": "No results", "query": args.query})); sys.exit(1)

    if getattr(args, "json"):
        print(json.dumps({"query": args.query, "results": results}, ensure_ascii=False, indent=2))
    else:
        print(fmt_results(results, args.urls_only))

if __name__ == "__main__":
    main()


path: searxng/docker-compose.yml
content: # Docker Compose for SearxNG
# Usage: cd searxng && docker-compose up -d
# Then configure ask-search: ask-search --searxng-url http://localhost:8080

version: '3.8'

services:
  searxng:
    image: searxng/searxng:latest
    container_name: searxng
    ports:
      - "127.0.0.1:8080:8080"
    environment:
      # Generate a secure secret key: python3 -c "import secrets; print(secrets.token_hex(32))"
      - SEARXNG_SECRET=${SEARXNG_SECRET:-your-secret-key-change-in-production}
      - SEARXNG_BASE_URL=http://localhost:8080
      # Enable JSON output for API access
      - SEARXNG_SETTINGS_SOURCE=docker
    volumes:
      - ./searxng.yml:/etc/searxng/settings.yml:ro
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080"]
      interval: 30s
      timeout: 10s
      retries: 3

# Usage:
# 1. Generate a secret key: python3 -c "import secrets; print(secrets.token_hex(32))"
# 2. Copy .env.example to .env and set SEARXNG_SECRET
# 3. Run: docker-compose up -d
# 4. Access SearxNG UI at http://localhost:8080
# 5. Configure ask-search: ask-search --searxng-url http://localhost:8080 --searxng-secret your-secret


path: searxng/searxng.yml
content: # Minimal SearxNG settings for Docker
# Enables JSON output for ask-search API access

general:
  debug: false
  instance_name: "ask-search SearxNG"

server:
  secret_key: "&{SEARXNG_SECRET}"
  bind_address: "0.0.0.0"
  port: 8080

ui:
  static_use_hash: true
  default_theme: simple

engines:
  - name: google
    engine: google
    shortcut: google

  - name: duckduckgo
    engine: duckduckgo
    shortcut: ddg

  - name: brave
    engine: brave
    shortcut: brave


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)