Skill: 1999azzar/search-cluster (version 3.5.1)

--- name: search-cluster description: Aggregated search aggregator using Google CSE, GNews RSS, Wikipedia, Reddit, and Scrapling. --- # Search Cluster (Industrial Standard v3.1) A multi-provider search aggregator designed for high-availability and security. ## Installation The scrapling provider requires a dedicated virtual environment. 1. Create a venv: python3 -m venv venv/scrapling 2. Install scrapling: venv/scrapling/bin/pip install scrapling 3. Provide the path to the venv binary in SCRAPLING_PYTHON_PATH. ## Security Posture - Subprocess Isolation: Query inputs are passed as arguments to stealth_fetch.py. - Strict TLS: Mandatory SSL verification on all providers. - Sanitization: Integrated native internal scrubber (Path Neutral). ## Requirements and Environment Declare these variables in your environment or vault: | Variable | Requirement | Description | |---|---|---| | GOOGLE_API_KEY | Optional | API Key for Google Custom Search. | | GOOGLE_CSE_ID | Optional | Search Engine ID for Google CSE. | | SCRAPLING_PYTHON_PATH | Optional | Path to the scrapling venv python binary. | | REDIS_HOST | Optional | Host for result caching. | | REDIS_PORT | Optional | Port for result caching (Default: 6379). | | SEARCH_USER_AGENT | Optional | Custom User-Agent string. | ## Providers - google: Official Google Custom Search. - wiki: Wikipedia OpenSearch API. - reddit: Reddit JSON search API. - gnews: Google News RSS aggregator. - scrapling: Headless stealth scraping (via DuckDuckGo). ## Included Scripts - scripts/search-cluster.py: Main entry point. - scripts/stealth_fetch.py: Scrapling fetcher (REQUIRED for scrapling provider). ## Workflow 1. Execute: scripts/search-cluster.py all "<query>" 2. Output is structured JSON with source, title, link, and sanitized snippet.

path: references/search-apis.md
content: # Search APIs Reference (Standard v3.0)

## Google Custom Search API
- **Endpoint**: `https://www.googleapis.com/customsearch/v1`
- **Requires**: `GOOGLE_API_KEY`, `GOOGLE_CSE_ID`
- **Purpose**: Official, high-quality search results.

## Wikipedia OpenSearch API
- **Endpoint**: `https://en.wikipedia.org/w/api.php`
- **Parameters**: `action=opensearch`, `format=json`, `namespace=0`
- **Purpose**: Fast, structured encyclopedia lookups.

## Reddit Search API
- **Endpoint**: `https://www.reddit.com/search.json`
- **Requires**: Proper `User-Agent` (handled by script).
- **Purpose**: Community discussions and real-time sentiment.

## Google News (GNews) RSS
- **Endpoint**: `https://news.google.com/rss/search`
- **Purpose**: Latest news headlines and tracking current events.

## DuckDuckGo Stealth (Scrapling)
- **Endpoint**: `https://duckduckgo.com/html/`
- **Requires**: `scrapling` package in a dedicated venv.
- **Purpose**: Privacy-respecting scraping for anti-bot bypass.


path: search-cluster.py
content: #!/usr/bin/env python3
import sys
import os
import json
import hashlib
import urllib.request
import urllib.parse
import ssl
import argparse
import concurrent.futures
import subprocess
import xml.etree.ElementTree as ET
import re

# Configuration (v3.5.1)
GOOGLE_API_KEY = os.getenv("GOOGLE_API_KEY")
GOOGLE_CSE_ID = os.getenv("GOOGLE_CSE_ID")
REDIS_HOST = os.getenv("REDIS_HOST")
REDIS_PORT = int(os.getenv("REDIS_PORT", 6379))
SCRAPLING_PYTHON = os.getenv("SCRAPLING_PYTHON_PATH", "python3")
# Corrected: Script is now in the same directory as this file
STEALTH_SCRIPT = os.path.join(os.path.dirname(__file__), "stealth_fetch.py")
USER_AGENT = os.getenv("SEARCH_USER_AGENT", "SearchCluster/3.5.1")

def internal_sanitize(text):
    if not text: return ""
    text = re.sub(r'ignore .*instructions|system override|you are now', '[REDACTED]', text, flags=re.I)
    text = "".join(ch for ch in text if ch.isprintable() or ch in ['\n', '\r', '\t'])
    return text.strip()

redis_client = None
if REDIS_HOST:
    try:
        import redis
        redis_client = redis.Redis(host=REDIS_HOST, port=REDIS_PORT, decode_responses=True, socket_connect_timeout=1)
    except: pass

def redis_set(key, value):
    if redis_client:
        try: redis_client.setex(key, 86400, value)
        except: pass

def redis_get(key):
    if redis_client:
        try: return redis_client.get(key)
        except: pass
    return None

def wiki_search(query):
    cache_key = f"search:wiki:{hashlib.md5(query.encode()).hexdigest()}"
    cached = redis_get(cache_key)
    if cached: return json.loads(cached)
    url = f"https://en.wikipedia.org/w/api.php?action=opensearch&search={urllib.parse.quote(query)}&limit=5&namespace=0&format=json"
    ctx = ssl.create_default_context()
    try:
        req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
        with urllib.request.urlopen(req, timeout=10, context=ctx) as response:
            data = json.loads(response.read().decode())
            results = []
            if isinstance(data, list) and len(data) > 3:
                for i in range(len(data[1])):
                    results.append({
                        "source": "wiki",
                        "title": internal_sanitize(data[1][i]),
                        "link": data[3][i],
                        "snippet": internal_sanitize(data[2][i])
                    })
            if results: redis_set(cache_key, json.dumps(results))
            return results
    except: return []

def reddit_search(query):
    cache_key = f"search:reddit:{hashlib.md5(query.encode()).hexdigest()}"
    cached = redis_get(cache_key)
    if cached: return json.loads(cached)
    url = f"https://www.reddit.com/search.json?q={urllib.parse.quote(query)}&limit=5"
    ctx = ssl.create_default_context()
    try:
        req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
        with urllib.request.urlopen(req, timeout=10, context=ctx) as response:
            data = json.loads(response.read().decode())
            results = []
            if "data" in data and "children" in data["data"]:
                for child in data["data"]["children"]:
                    post = child["data"]
                    results.append({
                        "source": "reddit",
                        "title": internal_sanitize(post.get("title")),
                        "link": f"https://reddit.com{post.get('permalink')}",
                        "snippet": internal_sanitize(f"r/{post.get('subreddit')} | {post.get('selftext', '')[:200]}")
                    })
            if results: redis_set(cache_key, json.dumps(results))
            return results
    except: return []

def scrapling_search(query):
    if not os.path.exists(SCRAPLING_PYTHON) or not os.path.exists(STEALTH_SCRIPT):
        return []
    try:
        result = subprocess.run([SCRAPLING_PYTHON, STEALTH_SCRIPT, query], capture_output=True, text=True, timeout=30)
        if result.returncode == 0:
            data = json.loads(result.stdout)
            for item in data:
                item["snippet"] = internal_sanitize(item.get("snippet", ""))
            return data
    except: pass
    return []

def gnews_search(query):
    url = f"https://news.google.com/rss/search?q={urllib.parse.quote(query)}&hl=en-US&gl=US&ceid=US:en"
    ctx = ssl.create_default_context()
    try:
        req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
        with urllib.request.urlopen(req, timeout=10, context=ctx) as response:
            root = ET.fromstring(response.read())
            results = []
            for item in root.findall(".//item")[:10]:
                results.append({
                    "source": "gnews",
                    "title": internal_sanitize(item.find("title").text),
                    "link": item.find("link").text,
                    "snippet": "Latest News via Google RSS"
                })
            return results
    except: return []

def google_search(query):
    if not GOOGLE_API_KEY or not GOOGLE_CSE_ID: return []
    url = f"https://www.googleapis.com/customsearch/v1?key={GOOGLE_API_KEY}&cx={GOOGLE_CSE_ID}&q={urllib.parse.quote(query)}"
    ctx = ssl.create_default_context()
    try:
        req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
        with urllib.request.urlopen(req, timeout=10, context=ctx) as response:
            data = json.loads(response.read().decode())
            results = []
            for item in data.get("items", []):
                results.append({
                    "source": "google", 
                    "title": internal_sanitize(item.get("title")), 
                    "link": item.get("link"), 
                    "snippet": internal_sanitize(item.get("snippet", ""))
                })
            return results
    except: return []

def search_all(query):
    results = []
    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        futures = {
            executor.submit(google_search, query): "google",
            executor.submit(wiki_search, query): "wiki",
            executor.submit(reddit_search, query): "reddit",
            executor.submit(gnews_search, query): "gnews",
            executor.submit(scrapling_search, query): "scrapling"
        }
        for f in concurrent.futures.as_completed(futures):
            try: results.extend(f.result())
            except: pass
    return results

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Aggregated Search v3.5.1")
    parser.add_argument("source", choices=["google", "wiki", "reddit", "gnews", "scrapling", "all"])
    parser.add_argument("query")
    args = parser.parse_args()
    try:
        if args.source == "all": res = search_all(args.query)
        elif args.source == "wiki": res = wiki_search(args.query)
        elif args.source == "reddit": res = reddit_search(args.query)
        elif args.source == "scrapling": res = scrapling_search(args.query)
        elif args.source == "gnews": res = gnews_search(args.query)
        else: res = google_search(args.query)
        print(json.dumps(res, indent=2))
    except Exception as e:
        print(json.dumps({"error": str(e)}))
        sys.exit(1)


path: skill.json
content: {
  "name": "search-cluster",
  "version": "3.5.1",
  "description": "Standard Aggregated Search Engine. Features Google, Wiki, Reddit, GNews, and Scrapling.",
  "author": "Mema",
  "homepage": "https://github.com/1999AZZAR/search-cluster-skill",
  "env_vars": [
    { "name": "GOOGLE_API_KEY", "description": "Google API Key (Optional)", "required": false },
    { "name": "GOOGLE_CSE_ID", "description": "Google CSE ID (Optional)", "required": false },
    { "name": "SCRAPLING_PYTHON_PATH", "description": "Path to scrapling python binary (Optional)", "required": false },
    { "name": "REDIS_HOST", "description": "Redis host for caching (Optional)", "required": false },
    { "name": "REDIS_PORT", "description": "Redis port (Optional, default 6379)", "required": false },
    { "name": "SEARCH_USER_AGENT", "description": "Custom User-Agent (Optional)", "required": false }
  ],
  "binaries": ["python3"],
  "python_dependencies": ["redis", "scrapling"]
}


path: stealth_fetch.py
content: import sys
import json
from scrapling import StealthyFetcher

def main():
    if len(sys.argv) < 2:
        print(json.dumps({"error": "No query provided"}))
        sys.exit(1)
        
    query = sys.argv[1]
    fetcher = StealthyFetcher()
    url = f"https://duckduckgo.com/html/?q={query.replace(' ', '+')}"
    
    try:
        page = fetcher.fetch(url)
        results = []
        for block in page.css('div.result'):
            title = block.css('a.result__a::text').get()
            link = block.css('a.result__a::attr(href)').get()
            snippet = block.css('a.result__snippet::text').get()
            if title and link:
                results.append({"source": "scrapling", "title": title, "link": link, "snippet": snippet})
        print(json.dumps(results))
    except Exception as e:
        print(json.dumps({"error": str(e)}))
        sys.exit(1)

if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: suspicious | SkillSpector: suspicious (CRITICAL)