Skill: staybased/peer-review (version 1.0.0)

--- name: peer-review description: | Multi-model peer review layer using local LLMs via Ollama to catch errors in cloud model output. Fan-out critiques to 2-3 local models, aggregate flags, synthesize consensus. Use when: validating trade analyses, reviewing agent output quality, testing local model accuracy, checking any high-stakes Claude output before publishing or acting on it. Don't use when: simple fact-checking (just search the web), tasks that don't benefit from multi-model consensus, time-critical decisions where 60s latency is unacceptable, reviewing trivial or low-stakes content. Negative examples: - "Check if this date is correct" → No. Just web search it. - "Review my grocery list" → No. Not worth multi-model inference. - "I need this answer in 5 seconds" → No. Peer review adds 30-60s latency. Edge cases: - Short text (<50 words) → Models may not find meaningful issues. Consider skipping. - Highly technical domain → Local models may lack domain knowledge. Weight flags lower. - Creative writing → Factual review doesn't apply well. Use only for logical consistency. version: "1.0" --- # Peer Review — Local LLM Critique Layer > **Hypothesis:** Local LLMs can catch ≥30% of real errors in cloud output with <50% false positive rate. --- ## Architecture ``` Cloud Model (Claude) produces analysis │ ▼ ┌────────────────────────┐ │ Peer Review Fan-Out │ ├────────────────────────┤ │ Drift (Mistral 7B) │──► Critique A │ Pip (TinyLlama 1.1B) │──► Critique B │ Lume (Llama 3.1 8B) │──► Critique C └────────────────────────┘ │ ▼ Aggregator (consensus logic) │ ▼ Final: original + flagged issues ``` --- ## Swarm Bot Roles | Bot | Model | Role | Strengths | |-----|-------|------|-----------| | **Drift** 🌊 | Mistral 7B | Methodical analyst | Structured reasoning, catches logical gaps | | **Pip** 🐣 | TinyLlama 1.1B | Fast checker | Quick sanity checks, low latency | | **Lume** 💡 | Llama 3.1 8B | Deep thinker | Nuanced analysis, catches subtle issues | --- ## Scripts | Script | Purpose | |--------|---------| | `scripts/peer-review.sh` | Send single input to all models, collect critiques | | `scripts/peer-review-batch.sh` | Run peer review across a corpus of samples | | `scripts/seed-test-corpus.sh` | Generate seeded error corpus for testing | ### Usage ```bash # Single file review bash scripts/peer-review.sh <input_file> [output_dir] # Batch review bash scripts/peer-review-batch.sh <corpus_dir> [results_dir] # Generate test corpus bash scripts/seed-test-corpus.sh [count] [output_dir] ``` Scripts live at `workspace/scripts/` — not bundled in skill to avoid duplication. --- ## Critique Prompt Template ``` You are a skeptical reviewer. Analyze the following text for errors. For each issue found, output JSON: {"category": "factual|logical|missing|overconfidence|hallucinated_source", "quote": "...", "issue": "...", "confidence": 0-100} If no issues found, output: {"issues": []} TEXT: --- {cloud_output} --- ``` --- ## Error Categories | Category | Description | Example | |----------|-------------|---------| | **factual** | Wrong numbers, dates, names | "Bitcoin launched in 2010" | | **logical** | Non-sequiturs, unsupported conclusions | "X is rising, therefore Y will fall" | | **missing** | Important context omitted | Ignoring a major counterargument | | **overconfidence** | Certainty without justification | "This will definitely happen" on 55% event | | **hallucinated_source** | Citing nonexistent sources | "According to a 2024 Reuters report..." | --- ## Discord Workflow 1. Post analysis to **#the-deep** (or #swarm-lab) 2. Drift, Pip, and Lume respond with independent critiques 3. Celeste synthesizes: deduplicates flags, weights by model confidence 4. If consensus (≥2 models agree) → flag is high-confidence 5. Final output posted with recommendation: `publish` | `revise` | `flag_for_human` --- ## Success Criteria | Outcome | TPR | FPR | Decision | |---------|-----|-----|----------| | **Strong pass** | ≥50% | <30% | Ship as default layer | | **Pass** | ≥30% | <50% | Ship as opt-in layer | | **Marginal** | 20–30% | 50–70% | Iterate on prompts, retest | | **Fail** | <20% | >70% | Abandon approach | ### Scoring Rules - Flag = **true positive** if it identifies a real error (even if explanation is imperfect) - Flag = **false positive** if flagged content is actually correct - Duplicate flags across models count once for TPR but inform consensus metrics --- ## Dependencies - Ollama running locally with models pulled: `mistral:7b`, `tinyllama:1.1b`, `llama3.1:8b` - `jq` and `curl` installed - Results stored in `experiments/peer-review-results/` --- ## Integration When peer review passes validation: - Package as Reef API endpoint: `POST /review` - Agents call before publishing any analysis - Configurable: model selection, consensus threshold, categories - Log all reviews to `#reef-logs` with TPR tracking


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: suspicious | SkillSpector: clean (LOW)