Skill: briefness/vryfik-skill (version 1.0.7)

--- name: searching-precisely description: Use when searching the web, documentation, or current information where token efficiency matters. Triggers on queries about API docs, current events, pricing, or any question requiring up-to-date online sources. Avoids reading entire pages when a targeted extract would suffice. # ── ClawHub Security Manifest ──────────────────────────────────────────────── # Declared so static analyzers can verify scripts match metadata. permissions: local_read: true # semantic-cache.js reads ~/.antigravity/search-cache/index.json local_write: true # semantic-cache.js writes ~/.antigravity/search-cache/index.json network: true # parallel-probe.js sends HTTP/HTTPS HEAD requests (no body upload) shell: true # agent runs node scripts via shell tool autonomous: false # must not be invoked with untrusted/unsanitized URL inputs disable-model-invocation: true # strictly enforces autonomous: false via OpenClaw policy host_provides: - web_search # actual web GET / search-API calls are delegated to the host agent # this skill handles pre-processing (intent, rewrite, cache) and # post-processing (probe, assemble, write-cache) only trust_model: "caller-trusted" # URLs are assumed to come from the host agent, not end-users metadata: openclaw: requires: bins: ["node"] # required runtime; scripts are invoked via shell tool io: cache_path: "~/.antigravity/search-cache/" # semantic-cache.js read/write scope # stores query text + result snippets (0o600) # created automatically on first write network_method: "HEAD only" # parallel-probe.js; no response body stored ssrf_protection: "two-layer" # 1. hostname pattern check # 2. DNS lookup → resolved-IP check (prevents DNS rebinding) security_notes: - "Cache files created with mode 0o600; cache dir with mode 0o700" - "Atomic cache writes: temp file + rename to prevent corruption" - "Network probes: HEAD only, 4 s timeout, max 5 concurrent, no body read" - "SSRF prevention layer 1: RFC-1918 + loopback + link-local hostname patterns rejected" - "SSRF prevention layer 2: hostname resolved to IP via dns.lookup(); IP re-checked against same patterns (prevents DNS rebinding)" - "No eval, no dynamic require, no code generation" - "Scripts invoked via shell tool only; all inputs are JSON-serialized strings" - "Cache may contain query text; ensure ~/.antigravity/search-cache/ access is restricted to owner" # ───────────────────────────────────────────────────────────────────────────── --- # Searching Precisely ## Overview Web search pipeline that minimizes token consumption via local intent classification, semantic caching, credibility validation, and streaming fragment assembly. **架构分工**： - **宿主 AI（Host Agent）**负责实际的网页 GET / Search API 调用，返回原始 fragments - **本 Skill 的脚本**负责前处理（intent 分类、query 改写、budget 控制、cache 查询）和后处理（credibility probe、stream 组装、cache 写入） **Core Rule:** Always check the semantic cache first. Only invoke web search on a cache miss. ## Pipeline Architecture ``` Query → [Intent Parser] → [Query Rewriter] → [Budget Controller] ↓ [Semantic Cache] ──hit──→ Return ↓ miss [Web Search] (≤1500 tok) ↓ [Parallel Credibility Probe] ↓ [Stream Assembler] → [Write Cache] ``` ## Instructions When this skill activates, execute the pipeline below **in order**. Exit early at any step that produces a final answer — do not run later steps unnecessarily. > **Note:** Replace `<placeholders>` with actual runtime values. All arguments must be valid JSON strings. --- ### Step 1 — Classify Intent Run via shell tool: ``` node scripts/intent-parser.js '<original_query>' ``` Extract `intent` and `confidence` from the JSON output. If `confidence < 0.5`, default to `intent = "web_search"` and continue. --- ### Step 2 — Initialize Budget ``` node scripts/budget-controller.js init ``` Keep the returned `state.remaining` value. Abort any later step that would exceed it. --- ### Step 3 — Check Semantic Cache ``` node scripts/semantic-cache.js check '{"query":"<original_query>","intent":"<intent>"}' ``` - `hit: true` and `similarity ≥ 0.85` → **return `result` to the user. Pipeline complete. Skip all remaining steps.** - `hit: false` → continue to Step 4. --- ### Step 4 — Rewrite Query ``` node scripts/query-rewriter.js '{"intent":"<intent>","query":"<original_query>"}' ``` Use the returned `subQueries` array (max 3) for web search. --- ### Step 5 — Web Search *(host agent)* Using your native `search_web` tool, search each sub-query from Step 4. Collect result URLs and content fragments. **Always perform live search on a cache miss — never fabricate results.** --- ### Step 6 — Validate Source Credibility Extract up to 5 unique source URLs from Step 5. Run: ``` node scripts/parallel-probe.js '{"sources":[{"url":"<url1>"},{"url":"<url2>"}]}' ``` - `verdict: "trust"` → use directly - `verdict: "verify"` → use with caution; flag in the answer - `available: false` → discard that source --- ### Step 7 — Score Credibility ``` node scripts/credibility-arbiter.js '{"results":[<probe_results_array>]}' ``` If **all** sources score `< 0.4`, discard everything and tell the user no reliable source was found. Do not assemble. --- ### Step 8 — Assemble Answer ``` node scripts/stream-assembler.js '{"fragments":[<trusted_fragments>],"query":"<original_query>"}' ``` Return the `answer` field to the user. If `coherenceScore < 0.5`, add a note that the result may be incomplete. --- ### Step 9 — Write Cache ``` node scripts/semantic-cache.js write '{"query":"<original_query>","intent":"<intent>","result":"<answer>"}' ``` This caches the result for future similar queries. Always run this step after a successful assembly. ## Search Decision Rules | Situation | Action | |-----------|--------| | Query answered by cache (similarity > 0.85) | Return cache, skip web search | | Query about API docs / official specs | Web search with known domain hint | | Query about current events / pricing / news | Web search, no cache assumed | | Query with ambiguous intent | Rewrite into 2-3 sub-queries, search each | ## Token Budget Defaults | Component | Budget | |-----------|--------| | Intent parse | 50 | | Cache check | 20 | | Web search | 1500 | | Credibility | 300 | | Assembly | 500 | | **Total** | **2370** | ## Common Mistakes - ❌ Calling web search before checking the semantic cache - ❌ Skipping credibility check for unknown domains - ❌ Assembling fragments from a single unverified source - ❌ Fetching full page content when a HEAD probe + summary suffices ## ClawHub Security Notes > All permission declarations, dependencies, I/O scopes, and security properties are > formally declared in the YAML frontmatter above (`permissions`, `dependencies`, `io`, > `security_notes`). Static analyzers should use those fields as the authoritative source. Summary per script: | Script | I/O | Network | Shell | |--------|-----|---------|-------| | `intent-parser.js` | none | none | none | | `query-rewriter.js` | none | none | none | | `stream-assembler.js` | none | none | none | | `budget-controller.js` | none | none | none | | `credibility-arbiter.js` | none | none | none | | `semantic-cache.js` | `~/.antigravity/search-cache/` R/W | none | none | | `parallel-probe.js` | reputation DB R (bundled) | HEAD only, no upload | none |

path: data/domain-reputation.json
content: {
  "version": "1.0.2",
  "updated": "2026-04-28",
  "scores": {
    "S": 1.0,
    "A": 0.8,
    "B": 0.6,
    "C": 0.4,
    "unknown": 0.3,
    "blocked": 0.0
  },
  "tiers": {
    "S": [
      "github.com",
      "stackoverflow.com",
      "mdn.mozilla.org",
      "developer.mozilla.org",
      "npmjs.com",
      "docs.python.org",
      "react.dev",
      "vuejs.org",
      "typescriptlang.org",
      "nodejs.org",
      "developer.apple.com",
      "developer.android.com",
      "docs.microsoft.com",
      "learn.microsoft.com",
      "go.dev",
      "rust-lang.org",
      "pkg.go.dev",
      "docs.rs",
      "cloud.google.com",
      "aws.amazon.com",
      "docs.docker.com",
      "kubernetes.io"
    ],
    "A": [
      "medium.com",
      "dev.to",
      "css-tricks.com",
      "smashingmagazine.com",
      "freecodecamp.org",
      "digitalocean.com",
      "web.dev",
      "hackernoon.com",
      "blog.logrocket.com",
      "codesandbox.io",
      "stackblitz.com"
    ],
    "B": [
      "reddit.com",
      "hashnode.com",
      "quora.com",
      "zhihu.com",
      "juejin.cn",
      "csdn.net",
      "cnblogs.com",
      "segmentfault.com"
    ],
    "C": [
      "w3schools.com"
    ],
    "blocked": []
  }
}


path: data/intent-patterns.json
content: {
  "version": "1.0.2",
  "file_search": {
    "rules": [
      { "keywords": ["in file", "in the file", "find file", "找文件", "哪个文件", "文件里", "文件中"], "weight": 3 },
      { "keywords": [".vue", ".ts", ".js", ".py", ".go", ".rs", ".json", ".yaml", ".yml"], "weight": 2 },
      { "keywords": ["path", "directory", "folder", "目录", "路径"], "weight": 1 }
    ]
  },
  "code_search": {
    "rules": [
      { "keywords": ["function", "class", "interface", "type", "component", "hook", "composable", "组件", "函数", "方法", "接口"], "weight": 3 },
      { "keywords": ["how to implement", "怎么实现", "如何实现", "用法", "示例", "example", "usage"], "weight": 2 },
      { "keywords": ["vue", "react", "typescript", "javascript", "python", "node", "api", "provide", "inject", "ref", "computed", "watch"], "weight": 2 },
      { "keywords": ["import", "export", "module", "require"], "weight": 1 }
    ]
  },
  "doc_search": {
    "rules": [
      { "keywords": ["documentation", "docs", "spec", "specification", "reference", "文档", "规范", "参考"], "weight": 3 },
      { "keywords": ["what is", "什么是", "定义", "概念", "definition"], "weight": 2 },
      { "keywords": ["mdn", "rfc", "w3c", "ecmascript", "proposal"], "weight": 2 }
    ]
  },
  "web_search": {
    "rules": [
      { "keywords": ["latest", "current", "today", "now", "最新", "现在", "今天", "近期"], "weight": 3 },
      { "keywords": ["price", "pricing", "cost", "费用", "价格"], "weight": 3 },
      { "keywords": ["news", "update", "release", "版本", "更新", "发布"], "weight": 2 },
      { "keywords": ["why", "compare", "vs", "versus", "对比", "为什么"], "weight": 1 }
    ]
  },
  "api_search": {
    "rules": [
      { "keywords": ["endpoint", "rest", "graphql", "openapi", "swagger", "接口", "api文档"], "weight": 3 },
      { "keywords": ["request", "response", "http", "get", "post", "put", "delete", "patch"], "weight": 2 },
      { "keywords": ["header", "body", "query param", "authorization", "auth", "token"], "weight": 2 }
    ]
  }
}


path: scripts/intent-parser.js
content: #!/usr/bin/env node
'use strict';

/**
 * Intent Parser — Local query classification
 *
 * ClawHub Security: NO network access. NO file system writes.
 * Pure synchronous computation. Deterministic output.
 *
 * Usage:
 *   node scripts/intent-parser.js "<query>"
 *   echo '{"query":"..."}' | node scripts/intent-parser.js
 *
 * Output JSON:
 *   { intent, confidence, scores, entities, lang, query }
 */

const path = require('path');
const fs   = require('fs');

// ---------------------------------------------------------------------------
// Load patterns (relative to this script)
// ---------------------------------------------------------------------------
const PATTERNS_PATH = path.join(__dirname, '..', 'data', 'intent-patterns.json');
const PATTERNS = JSON.parse(fs.readFileSync(PATTERNS_PATH, 'utf8'));

// ---------------------------------------------------------------------------
// Scoring
// ---------------------------------------------------------------------------
function scoreIntents(query) {
  const lower = query.toLowerCase();
  const scores = {};

  for (const [intent, spec] of Object.entries(PATTERNS)) {
    if (intent === 'version') continue;
    let score = 0;
    for (const rule of spec.rules) {
      if (rule.keywords.some(kw => lower.includes(kw))) {
        score += rule.weight;
      }
    }
    scores[intent] = score;
  }
  return scores;
}

// ---------------------------------------------------------------------------
// Entity extraction (no regex with catastrophic backtracking)
// ---------------------------------------------------------------------------
function extractEntities(query) {
  const lower = query.toLowerCase();

  // File extensions: simple split-based, no unbounded lookahead
  const fileTypes = query.split(/\s+/)
    .filter(t => /^\.[a-zA-Z]{2,5}$/.test(t))
    .slice(0, 5);

  // URLs: safe, anchored
  const urlMatches = query.match(/https?:\/\/[a-zA-Z0-9\-._~:/?#[\]@!$&'()*+,;=%]+/g) || [];

  // Technologies: keyword list lookup (no unbounded regex)
  const techList = [
    'vue', 'react', 'angular', 'svelte', 'typescript', 'javascript',
    'python', 'nodejs', 'node', 'css', 'html', 'api', 'rest',
    'graphql', 'sql', 'mongodb', 'redis', 'docker', 'kubernetes'
  ];
  const technologies = techList.filter(t => lower.includes(t));

  return { fileTypes, urls: urlMatches.slice(0, 3), technologies };
}

// ---------------------------------------------------------------------------
// Language detection
// ---------------------------------------------------------------------------
function detectLang(query) {
  return /[\u4e00-\u9fff]/.test(query) ? 'zh' : 'en';
}

// ---------------------------------------------------------------------------
// Main
// ---------------------------------------------------------------------------
function parse(query) {
  if (typeof query !== 'string' || query.trim().length === 0) {
    return { error: 'Invalid query: must be a non-empty string' };
  }

  // Sanitize: strip control characters, cap length
  const sanitized = query.replace(/[\x00-\x1f\x7f]/g, ' ').slice(0, 500).trim();

  const scores = scoreIntents(sanitized);
  const entries = Object.entries(scores).sort((a, b) => b[1] - a[1]);
  const [topIntent, topScore] = entries[0] || ['web_search', 0];
  const total = entries.reduce((s, [, v]) => s + v, 0);
  const confidence = total > 0 ? +(topScore / total).toFixed(3) : 0.2;

  return {
    intent:     topIntent,
    confidence,
    scores,
    entities:   extractEntities(sanitized),
    lang:       detectLang(sanitized),
    query:      sanitized
  };
}

// ---------------------------------------------------------------------------
// CLI
// ---------------------------------------------------------------------------
function main() {
  const arg = process.argv[2];

  if (arg) {
    // Try JSON first, fall back to raw string
    try {
      const parsed = JSON.parse(arg);
      console.log(JSON.stringify(parse(parsed.query ?? arg)));
    } catch {
      console.log(JSON.stringify(parse(arg)));
    }
    return;
  }

  // Stdin mode
  if (!process.stdin.isTTY) {
    let buf = '';
    process.stdin.setEncoding('utf8');
    process.stdin.on('data', c => { buf += c; });
    process.stdin.on('end', () => {
      try {
        const obj = JSON.parse(buf.trim());
        console.log(JSON.stringify(parse(obj.query ?? buf.trim())));
      } catch {
        console.log(JSON.stringify(parse(buf.trim())));
      }
    });
    return;
  }

  console.error('Usage: node scripts/intent-parser.js "<query>"');
  process.exit(1);
}

main();


path: scripts/semantic-cache.js
content: #!/usr/bin/env node
'use strict';

/**
 * Semantic Cache — Vector similarity cache backed by local JSON file
 *
 * ClawHub Security: FILE SYSTEM ONLY.
 * Read/write limited to CACHE_DIR (default: ~/.antigravity/search-cache/).
 * No network access. No eval. No dynamic code.
 *
 * Commands:
 *   node scripts/semantic-cache.js check '{"query":"...","intent":"..."}'
 *   node scripts/semantic-cache.js write '{"query":"...","intent":"...","result":"..."}'
 *   node scripts/semantic-cache.js evict   (remove expired/LRU entries)
 *   node scripts/semantic-cache.js stats
 */

const fs   = require('fs');
const path = require('path');
const os   = require('os');

// ---------------------------------------------------------------------------
// Config
// ---------------------------------------------------------------------------
const CACHE_DIR      = path.join(os.homedir(), '.antigravity', 'search-cache');
const CACHE_FILE     = path.join(CACHE_DIR, 'index.json');
const SIMILARITY_THRESHOLD = 0.82;   // minimum cosine similarity for a hit
const TTL_MS         = 24 * 60 * 60 * 1000;  // 24 hours
const MAX_ENTRIES    = 1000;
const MAX_RESULT_LEN = 8000;  // chars — prevents unbounded cache growth

// ---------------------------------------------------------------------------
// TF-IDF bag-of-words vector (no external deps)
// ---------------------------------------------------------------------------
const STOPWORDS = new Set([
  'a','an','the','is','are','was','were','be','been','being','have','has','had',
  'do','does','did','will','would','could','should','may','might','shall','can',
  'to','of','in','for','on','with','at','by','from','up','about','as','into',
  'through','during','including','until','before','after','since','without',
  'under','within','along','following','across','behind','beyond','plus',
  'except','and','or','but','not','no','nor','so','yet','both','either',
  'how','what','when','where','who','why','which','that','this','these','those',
  '的','了','在','是','我','有','和','就','不','人','都','一','一个','上','也','很',
  '到','说','要','去','你','会','着','没有','看','好','自己','这'
]);

function tokenize(text) {
  return text
    .toLowerCase()
    .replace(/[^\w\u4e00-\u9fff\s]/g, ' ')
    .split(/\s+/)
    .filter(t => t.length > 1 && !STOPWORDS.has(t));
}

function buildVector(tokens) {
  const freq = {};
  for (const t of tokens) freq[t] = (freq[t] || 0) + 1;
  return freq;
}

function cosineSimilarity(vecA, vecB) {
  const keysA = Object.keys(vecA);
  if (keysA.length === 0) return 0;

  let dot = 0, magA = 0, magB = 0;
  for (const k of keysA) {
    dot  += vecA[k] * (vecB[k] || 0);
    magA += vecA[k] ** 2;
  }
  for (const v of Object.values(vecB)) magB += v ** 2;

  if (magA === 0 || magB === 0) return 0;
  return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}

function queryToVector(query, intent) {
  const composite = `${intent} ${query}`;
  return buildVector(tokenize(composite));
}

// ---------------------------------------------------------------------------
// Cache I/O
// ---------------------------------------------------------------------------
function ensureCacheDir() {
  if (!fs.existsSync(CACHE_DIR)) {
    fs.mkdirSync(CACHE_DIR, { recursive: true, mode: 0o700 });
  }
}

function loadCache() {
  ensureCacheDir();
  if (!fs.existsSync(CACHE_FILE)) return [];
  try {
    const raw = fs.readFileSync(CACHE_FILE, 'utf8');
    return JSON.parse(raw);
  } catch {
    return [];
  }
}

function saveCache(entries) {
  ensureCacheDir();
  // Atomic write: write to temp file then rename
  const tmp = `${CACHE_FILE}.tmp`;
  fs.writeFileSync(tmp, JSON.stringify(entries, null, 0), { mode: 0o600 });
  fs.renameSync(tmp, CACHE_FILE);
}

// ---------------------------------------------------------------------------
// Operations
// ---------------------------------------------------------------------------
function checkCache(query, intent) {
  const entries = loadCache();
  const now     = Date.now();
  const qVec    = queryToVector(query, intent);
  let   best    = null;
  let   bestSim = 0;

  for (const entry of entries) {
    // Skip expired
    if (now - entry.ts > TTL_MS) continue;
    // Skip wrong intent
    if (entry.intent !== intent) continue;

    const sim = cosineSimilarity(qVec, entry.vec);
    if (sim > bestSim) {
      bestSim = sim;
      best    = entry;
    }
  }

  if (best && bestSim >= SIMILARITY_THRESHOLD) {
    return { hit: true, result: best.result, similarity: +bestSim.toFixed(4), age_ms: now - best.ts };
  }
  return { hit: false, similarity: +bestSim.toFixed(4) };
}

function writeCache(query, intent, result) {
  const entries = loadCache();
  const now     = Date.now();

  // Evict expired
  const active = entries.filter(e => now - e.ts <= TTL_MS);

  // LRU evict if at capacity
  if (active.length >= MAX_ENTRIES) {
    active.sort((a, b) => b.ts - a.ts);  // newest first
    active.splice(MAX_ENTRIES - 1);
  }

  active.push({
    query:  query.slice(0, 300),
    intent,
    vec:    queryToVector(query, intent),
    result: String(result).slice(0, MAX_RESULT_LEN),
    ts:     now
  });

  saveCache(active);
  return { written: true, totalEntries: active.length };
}

function evictCache() {
  const entries = loadCache();
  const now     = Date.now();
  const active  = entries.filter(e => now - e.ts <= TTL_MS);
  saveCache(active);
  return { removed: entries.length - active.length, remaining: active.length };
}

function statsCache() {
  const entries = loadCache();
  const now     = Date.now();
  const active  = entries.filter(e => now - e.ts <= TTL_MS);
  const byIntent = {};
  for (const e of active) byIntent[e.intent] = (byIntent[e.intent] || 0) + 1;
  return { total: entries.length, active: active.length, byIntent };
}

// ---------------------------------------------------------------------------
// CLI
// ---------------------------------------------------------------------------
function main() {
  const cmd = process.argv[2];
  const arg = process.argv[3];

  switch (cmd) {
    case 'check': {
      const { query, intent } = JSON.parse(arg);
      console.log(JSON.stringify(checkCache(query, intent)));
      break;
    }
    case 'write': {
      const { query, intent, result } = JSON.parse(arg);
      console.log(JSON.stringify(writeCache(query, intent, result)));
      break;
    }
    case 'evict':
      console.log(JSON.stringify(evictCache()));
      break;
    case 'stats':
      console.log(JSON.stringify(statsCache()));
      break;
    default:
      console.error('Usage: node scripts/semantic-cache.js <check|write|evict|stats> [json]');
      process.exit(1);
  }
}

main();


path: scripts/parallel-probe.js
content: #!/usr/bin/env node
'use strict';

/**
 * Parallel Probe — Concurrent HTTP HEAD checks for source availability
 *
 * ClawHub Security: OUTBOUND HTTP/HTTPS HEAD REQUESTS ONLY.
 * - Only HEAD method: no response body is read or stored.
 * - No user data is transmitted in requests.
 * - SSRF prevention (two layers):
 *     1. Hostname pattern check (blocks localhost, RFC-1918 literals, link-local)
 *     2. DNS resolution check (resolves hostname to IP, re-checks IP against patterns)
 *        This prevents DNS rebinding attacks where a public hostname resolves to a private IP.
 * - Timeout: 4 seconds per request.
 * - Max concurrent: 5 probes.
 *
 * Input  JSON: { sources: [{ url, domain? }] }
 * Output JSON: { results: [{ url, domain, available, statusCode, latencyMs, credibility }] }
 */

const https   = require('https');
const http    = require('http');
const dns     = require('dns').promises;
const url_mod = require('url');
const path    = require('path');
const fs      = require('fs');

// Load reputation for enrichment
const REP_PATH = path.join(__dirname, '..', 'data', 'domain-reputation.json');
const REP      = JSON.parse(fs.readFileSync(REP_PATH, 'utf8'));
const DOMAIN_TIER = {};
for (const [tier, domains] of Object.entries(REP.tiers)) {
  for (const d of domains) DOMAIN_TIER[d] = tier;
}

const TIMEOUT_MS     = 4000;
const MAX_PROBES     = 5;
const ALLOWED_PROTOS = ['http:', 'https:'];

// ---------------------------------------------------------------------------
// SSRF Prevention: block private / loopback / link-local addresses
// Covers: RFC 1918, loopback, link-local, IPv6 loopback/ULA/link-local
// ---------------------------------------------------------------------------
const BLOCKED_HOST_PATTERNS = [
  /^localhost$/i,
  /^127\./,                          // IPv4 loopback
  /^0\.0\.0\.0$/,                    // unspecified
  /^10\./,                           // RFC 1918 Class A
  /^172\.(1[6-9]|2\d|3[01])\./,     // RFC 1918 Class B
  /^192\.168\./,                     // RFC 1918 Class C
  /^169\.254\./,                     // link-local
  /^::1$/,                           // IPv6 loopback
  /^fc[0-9a-f]{2}:/i,                // IPv6 ULA
  /^fe80:/i,                         // IPv6 link-local
  /^0::/,                            // IPv6 unspecified
];

function isBlockedHost(hostname) {
  return BLOCKED_HOST_PATTERNS.some(p => p.test(hostname));
}

// ---------------------------------------------------------------------------
// Single HEAD probe — async for DNS resolution
// ---------------------------------------------------------------------------
async function probeOne(rawUrl) {
  const start = Date.now();

  let parsed;
  try {
    parsed = new url_mod.URL(rawUrl);
  } catch {
    return { url: rawUrl, available: false, error: 'invalid_url' };
  }

  // Only allow http/https
  if (!ALLOWED_PROTOS.includes(parsed.protocol)) {
    return { url: rawUrl, available: false, error: 'disallowed_protocol' };
  }

  // Layer 1: block by hostname pattern (fast path)
  if (isBlockedHost(parsed.hostname)) {
    return { url: rawUrl, available: false, error: 'blocked_host' };
  }

  // Layer 2: DNS resolution → IP-level check (prevents DNS rebinding bypass)
  let resolvedIp;
  try {
    const { address } = await dns.lookup(parsed.hostname);
    resolvedIp = address;
  } catch {
    return { url: rawUrl, available: false, error: 'dns_resolution_failed' };
  }
  if (isBlockedHost(resolvedIp)) {
    return { url: rawUrl, available: false, error: 'blocked_resolved_ip' };
  }

  const lib    = parsed.protocol === 'https:' ? https : http;
  const domain = parsed.hostname.replace(/^www\./, '');
  const tier   = DOMAIN_TIER[domain] || 'unknown';
  const score  = REP.scores[tier] || REP.scores.unknown;

  return new Promise(resolve => {
    const options = {
      method:   'HEAD',
      hostname: parsed.hostname,
      port:     parsed.port || (parsed.protocol === 'https:' ? 443 : 80),
      path:     parsed.pathname + parsed.search,
      timeout:  TIMEOUT_MS,
      headers:  {
        'User-Agent': 'search-skill-probe/1.0 (availability-check)'
      }
    };

    const req = lib.request(options, res => {
      const latencyMs = Date.now() - start;
      res.resume();  // discard any body (HEAD should have none, but be safe)
      resolve({
        url:        rawUrl,
        domain,
        tier,
        score,
        available:  res.statusCode < 400,
        statusCode: res.statusCode,
        latencyMs,
        verdict:    res.statusCode < 400 && score >= 0.6 ? 'trust' : 'verify'
      });
    });

    req.on('timeout', () => {
      req.destroy();
      resolve({ url: rawUrl, domain, available: false, error: 'timeout', latencyMs: TIMEOUT_MS });
    });

    req.on('error', err => {
      resolve({ url: rawUrl, domain, available: false, error: err.code || 'network_error' });
    });

    req.end();
  });
}

// ---------------------------------------------------------------------------
// Parallel probe with concurrency cap
// ---------------------------------------------------------------------------
async function probeAll(sources) {
  const limited = sources.slice(0, MAX_PROBES);
  const results = await Promise.allSettled(limited.map(s => probeOne(s.url || s)));
  return results.map(r => r.status === 'fulfilled' ? r.value : { available: false, error: 'rejected' });
}

// ---------------------------------------------------------------------------
// CLI
// ---------------------------------------------------------------------------
async function main() {
  const arg = process.argv[2];

  if (!arg) {
    console.error('Usage: node scripts/parallel-probe.js \'{"sources":[{"url":"https://vuejs.org"}]}\'');
    process.exit(1);
  }

  let input;
  try {
    input = JSON.parse(arg);
  } catch {
    console.log(JSON.stringify({ error: 'Invalid JSON input' }));
    process.exit(1);
  }

  const sources = Array.isArray(input.sources) ? input.sources : [];
  if (sources.length === 0) {
    console.log(JSON.stringify({ error: 'sources array is empty' }));
    process.exit(1);
  }

  const results = await probeAll(sources);
  const available = results.filter(r => r.available).length;

  console.log(JSON.stringify({
    results,
    summary: {
      total:      results.length,
      available,
      unavailable: results.length - available
    }
  }));
}

main().catch(err => {
  console.log(JSON.stringify({ error: err.message }));
  process.exit(1);
});


path: scripts/query-rewriter.js
content: #!/usr/bin/env node
'use strict';

/**
 * Query Rewriter — Expands a single query into targeted sub-queries
 *
 * ClawHub Security: NO network access. NO file system writes.
 * Pure synchronous computation. Deterministic output.
 *
 * Input  JSON: { intent, query, entities, lang }
 * Output JSON: { subQueries: string[], strategy: string, estimatedLayers: string[] }
 */

// ---------------------------------------------------------------------------
// Expansion templates per intent
// ---------------------------------------------------------------------------
const TEMPLATES = {
  code_search: [
    q => q,
    q => `${q} example`,
    q => `${q} implementation`,
    q => `how to use ${q}`,
    q => `${q} best practice`
  ],
  file_search: [
    q => q,
    q => `"${q}"`,          // exact phrase for ripgrep
    q => q.replace(/\s+/g, '_'),   // snake_case variant
    q => q.replace(/\s+/g, '-')    // kebab-case variant
  ],
  doc_search: [
    q => `${q} documentation`,
    q => `${q} reference`,
    q => `${q} API`,
    q => `${q} guide`
  ],
  web_search: [
    q => q,
    q => `${q} site:stackoverflow.com`,
    q => `${q} site:github.com`
  ],
  api_search: [
    q => `${q} endpoint`,
    q => `${q} REST API`,
    q => `${q} OpenAPI spec`,
    q => `${q} request response example`
  ]
};

// ---------------------------------------------------------------------------
// Strategy matrix: maps intent → recommended layer order
// ---------------------------------------------------------------------------
const STRATEGY_MAP = {
  file_search:  { strategy: 'l1_only',        estimatedLayers: ['l1'] },
  code_search:  { strategy: 'l1_then_l2',     estimatedLayers: ['l1', 'l2'] },
  doc_search:   { strategy: 'l2_then_l3',     estimatedLayers: ['l2', 'l3'] },
  api_search:   { strategy: 'l2_then_l3',     estimatedLayers: ['l2', 'l3'] },
  web_search:   { strategy: 'l3_only',        estimatedLayers: ['l3'] }
};

// ---------------------------------------------------------------------------
// Chinese → English keyword bridging for mixed-language queries
// ---------------------------------------------------------------------------
const ZH_EN_BRIDGE = {
  '组件': 'component',
  '函数': 'function',
  '方法': 'method',
  '接口': 'interface',
  '用法': 'usage',
  '示例': 'example',
  '实现': 'implementation',
  '文档': 'documentation',
  '参数': 'parameter',
  '返回值': 'return value'
};

function bridgeZhTerms(query) {
  let result = query;
  for (const [zh, en] of Object.entries(ZH_EN_BRIDGE)) {
    result = result.replace(new RegExp(zh, 'g'), `${zh} ${en}`);
  }
  return result;
}

// ---------------------------------------------------------------------------
// Main
// ---------------------------------------------------------------------------
function rewrite(input) {
  const { intent = 'web_search', query = '', lang = 'en' } = input;

  if (!query.trim()) {
    return { error: 'Empty query' };
  }

  // Bridge Chinese terms when lang is zh
  const base = lang === 'zh' ? bridgeZhTerms(query) : query;

  const templates = TEMPLATES[intent] || TEMPLATES.web_search;

  // Generate sub-queries, deduplicate, cap at 5
  const seen = new Set();
  const subQueries = templates
    .map(fn => fn(base).trim())
    .filter(q => {
      if (seen.has(q)) return false;
      seen.add(q);
      return true;
    })
    .slice(0, 5);

  const strategyInfo = STRATEGY_MAP[intent] || STRATEGY_MAP.web_search;

  return {
    subQueries,
    ...strategyInfo,
    lang,
    originalQuery: query
  };
}

// ---------------------------------------------------------------------------
// CLI
// ---------------------------------------------------------------------------
function main() {
  const arg = process.argv[2];

  if (arg) {
    try {
      console.log(JSON.stringify(rewrite(JSON.parse(arg))));
    } catch {
      console.log(JSON.stringify({ error: 'Invalid JSON input', received: arg }));
    }
    return;
  }

  if (!process.stdin.isTTY) {
    let buf = '';
    process.stdin.setEncoding('utf8');
    process.stdin.on('data', c => { buf += c; });
    process.stdin.on('end', () => {
      try {
        console.log(JSON.stringify(rewrite(JSON.parse(buf.trim()))));
      } catch {
        console.log(JSON.stringify({ error: 'Invalid JSON input' }));
      }
    });
    return;
  }

  console.error('Usage: node scripts/query-rewriter.js \'{"intent":"code_search","query":"Vue provide inject"}\'');
  process.exit(1);
}

main();


path: scripts/budget-controller.js
content: #!/usr/bin/env node
'use strict';

/**
 * Token Budget Controller — Stateless budget calculator
 *
 * ClawHub Security: NO network access. NO file system access.
 * Caller passes current state; script returns updated state.
 * Pure synchronous computation.
 *
 * Commands:
 *   node scripts/budget-controller.js init [totalBudget]
 *   node scripts/budget-controller.js check  '{"state":{...},"component":"l1_grep","amount":50}'
 *   node scripts/budget-controller.js consume '{"state":{...},"component":"l1_grep","amount":50}'
 *   node scripts/budget-controller.js report  '{"state":{...}}'
 */

// ---------------------------------------------------------------------------
// Default limits per component (tokens)
// ---------------------------------------------------------------------------
const DEFAULT_LIMITS = {
  intent_parse:   50,
  cache_check:    20,
  query_rewrite:  30,
  l1_grep:       100,
  l2_fragment:   400,
  l3_web:       1500,
  credibility:   300,
  assembly:      500
};

const DEFAULT_TOTAL = 4000;

// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
function sumConsumed(consumed) {
  return Object.values(consumed).reduce((s, v) => s + v, 0);
}

function initState(total = DEFAULT_TOTAL) {
  return {
    total: Number(total) || DEFAULT_TOTAL,
    limits: { ...DEFAULT_LIMITS },
    consumed: {},
    remaining: Number(total) || DEFAULT_TOTAL
  };
}

function checkBudget(state, component, amount) {
  if (!state || typeof state !== 'object') return { allowed: false, reason: 'Invalid state' };
  const limit   = state.limits?.[component];
  const already = state.consumed?.[component] || 0;
  const globalRemaining = state.total - sumConsumed(state.consumed || {});

  if (limit !== undefined && already + amount > limit) {
    return { allowed: false, reason: `Component limit exceeded (${already}+${amount} > ${limit})` };
  }
  if (globalRemaining < amount) {
    return { allowed: false, reason: `Global budget exhausted (need ${amount}, have ${globalRemaining})` };
  }
  return { allowed: true, globalRemaining: globalRemaining - amount };
}

function consumeBudget(state, component, amount) {
  const check = checkBudget(state, component, amount);
  if (!check.allowed) return { error: check.reason, state };

  const newConsumed = {
    ...state.consumed,
    [component]: (state.consumed[component] || 0) + amount
  };
  const totalConsumed = sumConsumed(newConsumed);

  return {
    state: {
      ...state,
      consumed: newConsumed,
      remaining: state.total - totalConsumed
    }
  };
}

function reportBudget(state) {
  const totalConsumed = sumConsumed(state.consumed || {});
  const breakdown = {};
  for (const [comp, limit] of Object.entries(state.limits || DEFAULT_LIMITS)) {
    breakdown[comp] = {
      limit,
      consumed: state.consumed?.[comp] || 0,
      remaining: limit - (state.consumed?.[comp] || 0)
    };
  }
  return {
    total: state.total,
    totalConsumed,
    totalRemaining: state.total - totalConsumed,
    utilizationPct: +((totalConsumed / state.total) * 100).toFixed(1),
    breakdown
  };
}

// ---------------------------------------------------------------------------
// CLI
// ---------------------------------------------------------------------------
function main() {
  const cmd = process.argv[2];
  const arg = process.argv[3];

  if (cmd === 'init') {
    console.log(JSON.stringify({ state: initState(arg) }));
    return;
  }

  if (!arg) {
    console.error('Usage: node scripts/budget-controller.js <init|check|consume|report> [json]');
    process.exit(1);
  }

  let input;
  try {
    input = JSON.parse(arg);
  } catch {
    console.log(JSON.stringify({ error: 'Invalid JSON input' }));
    process.exit(1);
  }

  switch (cmd) {
    case 'check':
      console.log(JSON.stringify(checkBudget(input.state, input.component, input.amount)));
      break;
    case 'consume':
      console.log(JSON.stringify(consumeBudget(input.state, input.component, input.amount)));
      break;
    case 'report':
      console.log(JSON.stringify(reportBudget(input.state)));
      break;
    default:
      console.log(JSON.stringify({ error: `Unknown command: ${cmd}` }));
      process.exit(1);
  }
}

main();


path: scripts/credibility-arbiter.js
content: #!/usr/bin/env node
'use strict';

/**
 * Credibility Arbiter — Domain reputation scoring (pure computation)
 *
 * ClawHub Security: NO network access. NO file system writes.
 * Reads domain-reputation.json once at startup. Pure synchronous scoring.
 *
 * Input  JSON: { sources: [{ url, domain?, content? }] }
 * Output JSON: { results: [{ domain, tier, score, verdict }], overallScore }
 */

const fs   = require('fs');
const path = require('path');
const url  = require('url');

// ---------------------------------------------------------------------------
// Load reputation data
// ---------------------------------------------------------------------------
const REP_PATH = path.join(__dirname, '..', 'data', 'domain-reputation.json');
const REP      = JSON.parse(fs.readFileSync(REP_PATH, 'utf8'));

// Build fast lookup: domain → tier
const DOMAIN_TIER = {};
for (const [tier, domains] of Object.entries(REP.tiers)) {
  for (const d of domains) DOMAIN_TIER[d] = tier;
}

// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
function extractDomain(rawUrl) {
  try {
    const parsed = new url.URL(rawUrl.startsWith('http') ? rawUrl : `https://${rawUrl}`);
    // Remove www. prefix for consistent lookup
    return parsed.hostname.replace(/^www\./, '');
  } catch {
    return rawUrl.toLowerCase().replace(/^www\./, '').split('/')[0];
  }
}

function scoreDomain(domain) {
  // Exact match first
  if (DOMAIN_TIER[domain] !== undefined) {
    const tier = DOMAIN_TIER[domain];
    return { tier, score: REP.scores[tier] };
  }

  // Subdomain fallback: check parent domain
  const parts = domain.split('.');
  if (parts.length > 2) {
    const parent = parts.slice(-2).join('.');
    if (DOMAIN_TIER[parent] !== undefined) {
      const tier = DOMAIN_TIER[parent];
      return { tier, score: REP.scores[tier] };
    }
  }

  // Blocked patterns
  if (REP.tiers.blocked.includes(domain)) {
    return { tier: 'blocked', score: 0 };
  }

  return { tier: 'unknown', score: REP.scores.unknown };
}

// Content heuristic scoring (no network — works on provided content snippet)
function scoreContent(content) {
  if (!content || typeof content !== 'string') return 1.0;  // neutral

  const lower = content.toLowerCase();
  let score = 1.0;

  // Positive signals
  if (/```|<code>|\.js|\.ts|\.py/.test(lower))   score += 0.1;  // has code
  if (/\d{4}/.test(lower))                        score += 0.05; // has dates
  if (lower.includes('example') || lower.includes('示例')) score += 0.05;

  // Negative signals
  if (lower.includes('click here') || lower.includes('buy now')) score -= 0.3;
  if ((lower.match(/!/g) || []).length > 5)       score -= 0.2;  // excessive exclamation

  return Math.min(1.2, Math.max(0.1, score));
}

// ---------------------------------------------------------------------------
// Dual-source agreement check
// ---------------------------------------------------------------------------
function dualSourceAgreement(results) {
  if (results.length < 2) return { agreed: null, reason: 'single_source' };

  const highQuality = results.filter(r => r.score >= 0.6);
  if (highQuality.length < 2) return { agreed: false, reason: 'insufficient_quality_sources' };

  // Simple agreement: both high-quality sources returned results
  return { agreed: true, sources: highQuality.map(r => r.domain) };
}

// ---------------------------------------------------------------------------
// Main
// ---------------------------------------------------------------------------
function arbitrate(sources) {
  if (!Array.isArray(sources) || sources.length === 0) {
    return { error: 'sources must be a non-empty array' };
  }

  const results = sources.slice(0, 10).map(src => {
    const domain    = src.domain || extractDomain(src.url || '');
    const { tier, score: domainScore } = scoreDomain(domain);
    const contentMult = scoreContent(src.content);
    const finalScore  = +(domainScore * contentMult).toFixed(3);

    return {
      domain,
      tier,
      domainScore,
      contentMultiplier: +contentMult.toFixed(3),
      finalScore,
      verdict: finalScore >= 0.7 ? 'trust' : finalScore >= 0.4 ? 'verify' : 'reject'
    };
  });

  const overallScore = results.length > 0
    ? +(results.reduce((s, r) => s + r.finalScore, 0) / results.length).toFixed(3)
    : 0;

  return {
    results,
    overallScore,
    dualSource: dualSourceAgreement(results),
    recommendation: overallScore >= 0.7 ? 'proceed'
      : overallScore >= 0.4             ? 'verify_before_use'
      :                                   'reject_find_better_sources'
  };
}

// ---------------------------------------------------------------------------
// CLI
// ---------------------------------------------------------------------------
function main() {
  const arg = process.argv[2];

  if (!arg) {
    console.error('Usage: node scripts/credibility-arbiter.js \'{"sources":[{"url":"..."}]}\'');
    process.exit(1);
  }

  try {
    const input = JSON.parse(arg);
    console.log(JSON.stringify(arbitrate(input.sources)));
  } catch (e) {
    console.log(JSON.stringify({ error: e.message }));
    process.exit(1);
  }
}

main();


path: scripts/stream-assembler.js
content: #!/usr/bin/env node
'use strict';

/**
 * Stream Assembler — Merges validated fragments into a coherent answer
 *
 * ClawHub Security: NO network access. NO file system writes.
 * Pure synchronous computation on caller-provided data.
 *
 * Input  JSON: { fragments: [...], query: string, lang?: "zh"|"en" }
 * Output JSON: { answer, coherenceScore, tokensEstimate, fragmentCount, warnings }
 */

// ---------------------------------------------------------------------------
// Token estimator (rough: 1 token ≈ 4 chars for English, 2 chars for Chinese)
// ---------------------------------------------------------------------------
function estimateTokens(text) {
  const chineseChars = (text.match(/[\u4e00-\u9fff]/g) || []).length;
  const otherChars   = text.length - chineseChars;
  return Math.ceil(chineseChars / 2 + otherChars / 4);
}

// ---------------------------------------------------------------------------
// Deduplication: remove fragments with > 80% overlapping content
// ---------------------------------------------------------------------------
function jaccard(a, b) {
  const setA = new Set(a.toLowerCase().split(/\s+/));
  const setB = new Set(b.toLowerCase().split(/\s+/));
  const intersection = [...setA].filter(x => setB.has(x)).length;
  const union = new Set([...setA, ...setB]).size;
  return union === 0 ? 0 : intersection / union;
}

function deduplicateFragments(fragments) {
  const kept = [];
  for (const frag of fragments) {
    const content = frag.content || frag.snippet || '';
    const isDuplicate = kept.some(k => {
      const kContent = k.content || k.snippet || '';
      return jaccard(content, kContent) > 0.8;
    });
    if (!isDuplicate) kept.push(frag);
  }
  return kept;
}

// ---------------------------------------------------------------------------
// Coherence scorer: measures semantic relatedness between consecutive fragments
// ---------------------------------------------------------------------------
function simpleTokenSet(text) {
  return new Set(
    text.toLowerCase()
      .replace(/[^\w\u4e00-\u9fff\s]/g, ' ')
      .split(/\s+/)
      .filter(t => t.length > 1)
  );
}

function overlapScore(textA, textB) {
  const setA = simpleTokenSet(textA);
  const setB = simpleTokenSet(textB);
  const inter = [...setA].filter(t => setB.has(t)).length;
  const min   = Math.min(setA.size, setB.size);
  return min === 0 ? 0 : inter / min;
}

function computeCoherence(fragments, query) {
  if (fragments.length === 0) return 0;
  if (fragments.length === 1) return 0.9;  // single fragment is always coherent

  let totalSim = 0;
  // Compare each fragment to the query + previous fragment
  for (let i = 0; i < fragments.length; i++) {
    const content = fragments[i].content || fragments[i].snippet || '';
    const queryScore = overlapScore(content, query);
    const prevScore  = i > 0
      ? overlapScore(content, fragments[i - 1].content || fragments[i - 1].snippet || '')
      : 1.0;
    totalSim += (queryScore * 0.6 + prevScore * 0.4);
  }

  return +(totalSim / fragments.length).toFixed(3);
}

// ---------------------------------------------------------------------------
// Format header for each fragment
// ---------------------------------------------------------------------------
function formatFragmentHeader(frag, idx, lang) {
  if (frag.file) {
    const lineInfo = frag.line ? `:${frag.line}` : '';
    return lang === 'zh'
      ? `**[${idx + 1}] 来源：${frag.file}${lineInfo}**`
      : `**[${idx + 1}] Source: ${frag.file}${lineInfo}**`;
  }
  if (frag.url || frag.domain) {
    return lang === 'zh'
      ? `**[${idx + 1}] 来源：${frag.domain || frag.url}**`
      : `**[${idx + 1}] Source: ${frag.domain || frag.url}**`;
  }
  return `**[${idx + 1}]**`;
}

// ---------------------------------------------------------------------------
// Main assembler
// ---------------------------------------------------------------------------
function assemble(input) {
  const { fragments = [], query = '', lang = 'en' } = input;

  if (fragments.length === 0) {
    return { answer: '', coherenceScore: 0, tokensEstimate: 0, fragmentCount: 0, warnings: ['no_fragments'] };
  }

  // Deduplicate
  const unique = deduplicateFragments(fragments);
  const warnings = [];
  if (unique.length < fragments.length) {
    warnings.push(`deduped_${fragments.length - unique.length}_fragments`);
  }

  // Score coherence before assembly
  const coherenceScore = computeCoherence(unique, query);
  if (coherenceScore < 0.3) warnings.push('low_coherence_consider_requery');

  // Build answer string (streaming-ready: each section is a complete block)
  const sections = unique.map((frag, i) => {
    const header  = formatFragmentHeader(frag, i, lang);
    const content = (frag.content || frag.snippet || '').trim().slice(0, 2000);
    return `${header}\n\n${content}`;
  });

  const answer = sections.join('\n\n---\n\n');
  const tokensEstimate = estimateTokens(answer);

  if (tokensEstimate > 3000) warnings.push('answer_may_be_too_long');

  return {
    answer,
    coherenceScore,
    tokensEstimate,
    fragmentCount: unique.length,
    lang,
    warnings
  };
}

// ---------------------------------------------------------------------------
// CLI
// ---------------------------------------------------------------------------
function main() {
  const arg = process.argv[2];

  if (!arg) {
    console.error('Usage: node scripts/stream-assembler.js \'{"fragments":[...],"query":"..."}\'');
    process.exit(1);
  }

  try {
    const input = JSON.parse(arg);
    console.log(JSON.stringify(assemble(input)));
  } catch (e) {
    console.log(JSON.stringify({ error: e.message }));
    process.exit(1);
  }
}

main();


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)