Skill: saikatkumardey/searxng-selfhost (version 1.0.1)

--- name: searxng description: Self-hosted web search aggregator for OpenClaw agents. Use this skill to (1) install SearXNG on a VPS/server so the agent can search the web without API keys, or (2) run web searches using an existing local SearXNG instance. Covers installation, configuration, the search.py CLI tool, and fallback behaviour. Use when the agent needs web search capability, when setting up a new OpenClaw instance, or when diagnosing search failures. --- # SearXNG SearXNG is a self-hosted search aggregator that queries Google, Bing, Brave, Startpage, DuckDuckGo, and Wikipedia simultaneously. No API keys required. Results returned as JSON. ## Quick start (already installed) ```bash python3 scripts/search.py "your query" # human-readable python3 scripts/search.py "your query" --json # JSON (for parsing) python3 scripts/search.py "query" --count 5 --json # limit + JSON ``` Place `search.py` anywhere convenient — typically `tools/search.py` in the workspace. For detailed usage patterns and service management: see `references/usage.md`. ## Installation (new instance) Run as root on Ubuntu 22.04/24.04: ```bash bash scripts/install_searxng.sh ``` This installs SearXNG, creates a `searxng` system user, writes `/etc/searxng/settings.yml`, and starts a systemd service on `http://127.0.0.1:8888`. Verify: ```bash curl 'http://127.0.0.1:8888/search?q=test&format=json' | python3 -m json.tool | head -20 systemctl status searxng ``` ## Connecting search.py to the instance `search.py` targets `http://127.0.0.1:8888` by default. If the port differs, update `SEARXNG_URL` at the top of the script. ## Fallback If SearXNG is down, `search.py` falls back to Wikipedia + GitHub APIs automatically. No action needed — results still return, just from narrower sources. ## Troubleshooting | Symptom | Fix | |---------|-----| | `[SearXNG unavailable]` in stderr | `systemctl restart searxng` | | Port conflict on 8888 | Change `port:` in `/etc/searxng/settings.yml` + update `SEARXNG_URL` in script | | Empty results from all engines | Check `/etc/searxng/settings.yml` engines block; restart service | | Connection refused | Service not running — `systemctl start searxng` |

path: references/usage.md
content: # SearXNG Search — Usage Reference

## Basic usage

```bash
python3 /path/to/search.py "query"                  # 10 results, human-readable
python3 /path/to/search.py "query" --count 5        # limit results
python3 /path/to/search.py "query" --json           # JSON output (for parsing)
python3 /path/to/search.py "query" --count 5 --json # both
```

## Result format

Each result has:
- `title` — page/document title
- `url` — full URL
- `snippet` — up to 200 chars of content
- `engine` — which engine returned it (google, bing, brave, startpage, duckduckgo, wikipedia)

## Fallback behaviour

If SearXNG is unavailable (service down, wrong port), the script falls back automatically to:
1. Wikipedia API (always works, no IP restrictions)
2. GitHub Search API (10 req/min unauthenticated)

Fallback is logged to stderr. Results still return — just fewer, narrower sources.

## Service management

```bash
systemctl status searxng        # check status
systemctl restart searxng       # restart
journalctl -u searxng -f        # live logs
```

## Health check

```bash
curl 'http://127.0.0.1:8888/search?q=test&format=json' | python3 -m json.tool | head -30
```

## Configuration

Config lives at `/etc/searxng/settings.yml`.
To add/remove engines, edit the `engines:` block and restart.

Port default: `8888`. To change, update `settings.yml` + update `SEARXNG_URL` in `search.py`.

## Common query patterns

| Goal | Query example |
|------|---------------|
| Current events | "topic site:reuters.com OR site:bbc.com" |
| GitHub repos | "project language python" — then follow with GitHub API for details |
| Docs lookup | "function name site:docs.python.org" |
| Price/availability | product name + "price" or "review 2025" |
| Technical errors | paste the exact error message in quotes |

## Resource usage

~85MB RAM. Negligible CPU at rest. Safe to run on a 2GB VPS alongside other services.


path: scripts/install_searxng.sh
content: #!/bin/bash
# SearXNG install script for OpenClaw agents
# Installs SearXNG as a local search aggregator on port 8888
# Tested on Ubuntu 22.04/24.04

set -e

SEARXNG_HOME="/usr/local/searxng"
SEARXNG_PORT=8888
SEARXNG_USER="searxng"

echo "[1/6] Installing system dependencies..."
apt-get update -qq
apt-get install -y -qq git python3-pip

# Install uv via pip if not present
if ! command -v uv &>/dev/null; then
  pip3 install -q uv --break-system-packages
fi

echo "[2/6] Creating searxng user and directories..."
id -u $SEARXNG_USER &>/dev/null || /usr/sbin/useradd -r -d $SEARXNG_HOME -s /bin/false $SEARXNG_USER
mkdir -p $SEARXNG_HOME
chown $SEARXNG_USER:$SEARXNG_USER $SEARXNG_HOME

echo "[3/6] Cloning SearXNG..."
if [ -d "$SEARXNG_HOME/searxng-src" ]; then
  echo "  Already cloned — skipping"
else
  git clone https://github.com/searxng/searxng "$SEARXNG_HOME/searxng-src" --depth=1
fi

echo "[4/6] Installing dependencies with uv..."
cd "$SEARXNG_HOME/searxng-src"
uv venv "$SEARXNG_HOME/searx-venv" --python python3
uv pip install --python "$SEARXNG_HOME/searx-venv/bin/python" \
  -r "$SEARXNG_HOME/searxng-src/requirements.txt"

echo "[5/6] Writing config..."
mkdir -p /etc/searxng
cat > /etc/searxng/settings.yml << EOF
use_default_settings: true
server:
  bind_address: "127.0.0.1"
  port: ${SEARXNG_PORT}
  secret_key: "$(python3 -c 'import secrets; print(secrets.token_hex(32))')"
  limiter: false
  public_instance: false
search:
  formats:
    - html
    - json
engines:
  - name: google
    engine: google
    shortcut: g
  - name: bing
    engine: bing
    shortcut: bi
  - name: brave
    engine: brave
    shortcut: br
  - name: startpage
    engine: startpage
    shortcut: sp
  - name: duckduckgo
    engine: duckduckgo
    shortcut: ddg
  - name: wikipedia
    engine: wikipedia
    shortcut: wp
EOF
chown -R $SEARXNG_USER:$SEARXNG_USER /etc/searxng

echo "[6/6] Installing systemd service..."
cat > /etc/systemd/system/searxng.service << EOF
[Unit]
Description=SearXNG (local search aggregator for OpenClaw)
After=network.target

[Service]
Type=simple
User=${SEARXNG_USER}
Environment=SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml
Environment=PYTHONPATH=${SEARXNG_HOME}/searxng-src
ExecStart=${SEARXNG_HOME}/searx-venv/bin/python -m searx.webapp
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable searxng
systemctl restart searxng

echo ""
echo "SearXNG installed and running."
echo "Health check: curl 'http://127.0.0.1:${SEARXNG_PORT}/search?q=test&format=json'"
echo "Logs: journalctl -u searxng -f"


path: scripts/search.py
content: #!/usr/bin/env python3
"""
Multi-backend search for agents.
Primary: local SearXNG instance (Google, Bing, Brave, Startpage, DDG, Wikipedia all in one)
Fallback: direct Wikipedia + GitHub APIs

Usage: search.py "your query" [--count N] [--json]
"""

import sys
import json
import re
import urllib.parse
import urllib.request

SEARXNG_URL = "http://127.0.0.1:8888"
HEADERS = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36',
    'Accept': 'application/json',
}

def search_searxng(query, count=10):
    """Local SearXNG instance - aggregates Google, Bing, Brave, Startpage, DDG, Wikipedia"""
    url = f"{SEARXNG_URL}/search?q={urllib.parse.quote(query)}&format=json"
    req = urllib.request.Request(url, headers=HEADERS)
    try:
        with urllib.request.urlopen(req, timeout=15) as r:
            d = json.loads(r.read())
            return [{'title': x.get('title', ''), 
                     'url': x.get('url', ''),
                     'snippet': x.get('content', '')[:200],
                     'engine': x.get('engine', '?')} for x in d.get('results', [])[:count]]
    except Exception as e:
        print(f"  [SearXNG unavailable: {e}]", file=sys.stderr)
        return None  # Signal to use fallbacks

def search_wikipedia(query, count=5):
    """Wikipedia API - always works"""
    url = f"https://en.wikipedia.org/w/api.php?action=query&list=search&srsearch={urllib.parse.quote(query)}&srlimit={count}&format=json"
    req = urllib.request.Request(url, headers=HEADERS)
    try:
        with urllib.request.urlopen(req, timeout=8) as r:
            d = json.loads(r.read())
            return [{'title': x['title'], 
                     'url': f"https://en.wikipedia.org/wiki/{urllib.parse.quote(x['title'].replace(' ', '_'))}",
                     'snippet': re.sub(r'<[^>]+>', '', x.get('snippet', '')),
                     'engine': 'wikipedia'} for x in d['query']['search'][:count]]
    except: return []

def search_github(query, count=5):
    """GitHub Search API - 10 req/min unauthenticated"""
    url = f"https://api.github.com/search/repositories?q={urllib.parse.quote(query)}&per_page={count}"
    req = urllib.request.Request(url, headers={**HEADERS, 'Accept': 'application/vnd.github.v3+json'})
    try:
        with urllib.request.urlopen(req, timeout=8) as r:
            d = json.loads(r.read())
            return [{'title': x['full_name'], 
                     'url': x['html_url'],
                     'snippet': (x.get('description') or '')[:200],
                     'engine': 'github'} for x in d.get('items', [])[:count]]
    except: return []

def search(query, count=10):
    """Search via SearXNG, fall back to direct APIs"""
    results = search_searxng(query, count)
    if results is not None:
        return results
    
    # Fallback: direct APIs
    print("  [Falling back to Wikipedia + GitHub]", file=sys.stderr)
    wp = search_wikipedia(query, count)
    gh = search_github(query, min(5, count))
    return (wp + gh)[:count]

if __name__ == '__main__':
    if len(sys.argv) < 2:
        print("Usage: search.py 'query' [--count N] [--json]")
        sys.exit(1)
    
    query = sys.argv[1]
    count = 10
    as_json = '--json' in sys.argv
    
    for i, arg in enumerate(sys.argv):
        if arg == '--count' and i + 1 < len(sys.argv):
            count = int(sys.argv[i + 1])
    
    results = search(query, count)
    
    if as_json:
        print(json.dumps(results, indent=2))
    else:
        if not results:
            print("No results found.")
        else:
            print(f"Found {len(results)} results:\n")
            for i, r in enumerate(results, 1):
                print(f"{i}. [{r['engine']}] {r['title']}")
                print(f"   {r['url']}")
                if r.get('snippet'):
                    print(f"   {r['snippet'][:150]}")
                print()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)