Skill: dacervera/quorum (version 0.7.3)

--- name: quorum description: Multi-agent validation framework — 6 independent AI critics evaluate artifacts against rubrics with evidence-grounded findings. metadata: {"openclaw":{"requires":{"bins":["python3","pip"],"env":["ANTHROPIC_API_KEY","OPENAI_API_KEY"]},"install":[{"id":"clone-repo","kind":"shell","command":"git clone https://github.com/SharedIntellect/quorum.git /tmp/quorum-install && cd /tmp/quorum-install/reference-implementation && pip install -r requirements.txt","label":"Clone Quorum repo and install Python dependencies"}],"source":"https://github.com/SharedIntellect/quorum"}} --- # Quorum — Multi-Agent Validation Quorum validates AI agent outputs by spawning multiple independent critics that evaluate artifacts against rubrics. Every criticism must cite evidence. You get a structured verdict. ## Quick Start Clone the repository and install: ```bash git clone https://github.com/SharedIntellect/quorum.git cd quorum/reference-implementation pip install -r requirements.txt ``` Run a quorum check on any file: ```bash python -m quorum.cli run --target <path-to-artifact> --rubric <rubric-name> ``` ### Built-in Rubrics - `research-synthesis` — Research reports, literature reviews, technical analyses - `agent-config` — Agent configurations, YAML specs, system prompts - `python-code` — Python source files (25 criteria, PC-001–PC-025; auto-detected on `.py` files) ### Depth Profiles - `quick` — 2 critics (correctness, completeness) + pre-screen, ~5-10 min - `standard` — 4 active (correctness, completeness, security + tester) + pre-screen, ~15-30 min (default) - `thorough` — 5 active (+ code_hygiene) + pre-screen + fix loops, ~30-60 min *†Cross-Consistency requires `--relationships` flag with a relationships manifest.* All depth profiles include the deterministic pre-screen (10 checks: credentials, PII, syntax errors, broken links, TODOs, and more) before any LLM critic runs. ### Examples ```bash # Validate a research report quorum run --target my-report.md --rubric research-synthesis # Quick check (faster, fewer critics) quorum run --target my-report.md --rubric research-synthesis --depth quick # Batch: validate all markdown files in a directory quorum run --target ./docs/ --pattern "*.md" --rubric research-synthesis # Cross-artifact consistency check quorum run --target ./src/ --relationships quorum-relationships.yaml --depth standard # Use a custom rubric quorum run --target my-spec.md --rubric ./my-rubric.json # List available rubrics quorum rubrics list # Initialize config interactively quorum config init ``` ## Configuration On first run, Quorum prompts for your preferred models and writes `quorum-config.yaml`. You can also create it manually: ```yaml models: tier_1: anthropic/claude-sonnet-4-6 # Judgment roles tier_2: anthropic/claude-sonnet-4-6 # Evaluation roles depth: standard ``` Set your API key: ```bash export ANTHROPIC_API_KEY=sk-ant-... # or export OPENAI_API_KEY=sk-... ``` ## Output Quorum produces a structured verdict: - **PASS** — No significant issues found - **PASS_WITH_NOTES** — Minor issues, artifact is usable - **REVISE** — High/critical issues that need rework before proceeding - **REJECT** — Unfixable problems; restart required Exit codes: `0` = PASS/PASS_WITH_NOTES, `1` = error, `2` = REVISE/REJECT. Each finding includes: severity (CRITICAL/HIGH/MEDIUM/LOW), evidence citations pointing to specific locations in the artifact, and remediation suggestions. The run directory contains `prescreen.json`, per-critic finding JSONs, `verdict.json`, and a human-readable `report.md`. ## More Information - [SPEC.md](https://github.com/SharedIntellect/quorum/blob/main/SPEC.md) — Full architectural specification - [MODEL_REQUIREMENTS.md](https://github.com/SharedIntellect/quorum/blob/main/docs/getting-started/MODEL_REQUIREMENTS.md) — Supported models and tiers - [CONFIG_REFERENCE.md](https://github.com/SharedIntellect/quorum/blob/main/docs/configuration/CONFIG_REFERENCE.md) — All configuration options - [FOR_BEGINNERS.md](https://github.com/SharedIntellect/quorum/blob/main/docs/getting-started/FOR_BEGINNERS.md) — New to agent validation? Start here --- > ⚖️ **LICENSE** — Not part of the operational specification above. > This file is part of [Quorum](https://github.com/SharedIntellect/quorum). > Copyright 2026 SharedIntellect. MIT License. > See [LICENSE](https://github.com/SharedIntellect/quorum/blob/main/LICENSE) for full terms.

path: CLAUDE.md
content: # CLAUDE.md — Agent Instructions for Quorum

This file provides context for Claude Code and other AI agents working on this repository.

## Project Overview

Quorum is a multi-critic quality assurance framework. The reference implementation lives in `reference-implementation/` and is published on PyPI as `quorum-validator`.

## Current State

- **6 of 9 critics shipped**: Correctness, Completeness, Security, Code Hygiene, Cross-Consistency (`--relationships` flag required), Tester (L1 + L2)
- **3 planned**: Architecture, Delegation, Style
- **Version**: See `critic-status.yaml` for the canonical version and critic status
- **Tests**: `cd reference-implementation && pytest tests/ -v` (exclude `test_e2e_smoke.py` and `test_performance.py` for offline runs)

## Shipping Checklist — MANDATORY

**When shipping any feature or critic to main, follow `SHIPPING.md` in the repo root.**

Key steps:
1. Update `critic-status.yaml` (the single source of truth for critic/feature status)
2. Bump version in `critic-status.yaml` AND `reference-implementation/pyproject.toml`
3. Run `python tools/validate-docs.py` — fix all findings before merging
4. Run `tools/boundary-scan.sh` (located in the workspace, not this repo) — ensure no private paths leak
5. Update `docs/CHANGELOG.md` with a full release entry
6. All doc files that reference critic counts or status markers must match the manifest

**Why this matters:** We've shipped features with 10+ doc files still claiming old critic counts. The manifest + validation script + CI gate exists to prevent that. Don't skip it.

## Key Files

| File | Purpose |
|------|---------|
| `critic-status.yaml` | Single source of truth for shipped critics and features |
| `SHIPPING.md` | Full shipping checklist |
| `tools/validate-docs.py` | Automated doc validation (reads manifest, flags stale references) |
| `SPEC.md` | Framework specification |
| `reference-implementation/` | Python CLI implementation |
| `docs/CHANGELOG.md` | Release history |

## Conventions

- **Evidence grounding**: Every critic finding must cite specific locus (file, line, excerpt). No hand-waving.
- **Test coverage**: New critics need both unit tests and integration tests. Target: 20+ tests per critic.
- **Status markers**: Use ✅ Shipped or 🔜 Planned in docs. Update `critic-status.yaml` first, then docs.
- **Boundary scan**: No private file paths, internal names, or credentials in public files. `quorum-runs/` is gitignored.


path: CONTRIBUTING.md
content: # Contributing to Quorum

Thank you for your interest in contributing to Quorum. This document covers how to get involved.

## Ways to Contribute

### 🔍 Submit Custom Rubrics

The rubric system is Quorum's most extensible surface. If you've built a rubric for a domain we don't cover yet, we'd love to include it.

**How:**
1. Create a JSON rubric following the format in `examples/rubrics/`
2. Test it against at least one real artifact
3. Open a PR adding it to `examples/rubrics/` with a brief description

**Good rubric contributions:**
- New domains (code review, documentation, API design, data pipelines)
- Specialized variants (security-focused, performance-focused)
- Industry-specific criteria (healthcare, finance, government)

### 🔗 Submit Cross-Artifact Relationship Manifests

Cross-artifact manifests are a new and growing contribution surface. If you have a project structure where multiple files have declared relationships (spec/implementation, schema/consumer, config/profile), contributing a well-structured manifest example helps the community understand how to use `--relationships` effectively.

**How:**
1. Create a `quorum-relationships.yaml` for a real or representative project
2. Test it with `quorum run --relationships` against the described files
3. Open a PR adding it to `examples/relationships/` with a description of the project structure and what consistency checks it enables

**Good manifest contributions:**
- Real-world project patterns (API spec ↔ handler, OpenAPI schema ↔ consumer)
- Framework-specific patterns (FastAPI, Django, OpenClaw swarms)
- Documentation consistency patterns (README ↔ code, SPEC ↔ implementation)

### 🐛 Bug Reports

Found an issue? Open a GitHub Issue with:
- What you were validating (artifact type, depth profile)
- What you expected vs. what happened
- Model and tier configuration
- Any relevant log output

### 💡 Feature Proposals

Have an idea for a new critic, aggregation strategy, or rubric feature? Open a Discussion thread. We're particularly interested in:
- New critic specializations
- Alternative aggregation strategies
- Integration patterns with specific agent frameworks
- Cost optimization techniques

### 📝 Documentation

Improvements to docs, tutorials, and examples are always welcome. If something was confusing when you first used Quorum, it's probably confusing for others too.

### 🔬 External Reviews

Run Quorum against your own system and share the results. See `docs/reviews/EXTERNAL_REVIEWS.md` for how to submit.

## Pull Request Process

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/your-feature`)
3. Make your changes
4. Ensure any new rubrics include at least one example validation
5. Open a PR with a clear description of what changed and why

## Code of Conduct

This project follows the [Contributor Covenant Code of Conduct](https://www.contributor-covenant.org/version/2/1/code_of_conduct/). By participating, you agree to uphold this standard.

## Questions?

Open a Discussion thread or reach out on X: [@Cervera](https://twitter.com/Cervera) or [@AkkariNova](https://twitter.com/AkkariNova).

---

*Quorum is maintained by Daniel Cervera and Akkari at SharedIntellect.*


path: README.md
content: <p align="center">
  <img src="/branding/github/dark_qu_gh.jpg" alt="Quorum" width="900">
</p>

<p align="center">
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-2ba4c8.svg" alt="MIT License"></a>
  <img src="https://img.shields.io/badge/platform-OpenClaw-2ba4c8" alt="Platform: OpenClaw">
  <img src="https://img.shields.io/badge/status-v0.7.2-2ba4c8" alt="Status: v0.7.2">
  <a href="https://clawhub.ai/dacervera/quorum"><img src="https://img.shields.io/badge/ClawHub-dacervera%2Fquorum-2ba4c8" alt="Available on ClawHub"></a>
</p>

---

## Hey. I'm Quorum. 🦞

You built something with your AI agent. A research report. A config. A codebase. Maybe a whole swarm produced it — five agents researching, synthesizing, writing — and now you're staring at the output wondering:

*"How do I know this is actually right?"*

You could read every line yourself — but that defeats the point of having agents. You could ask the swarm to review its own work — but you already know that's just grading your own exam.

**That's where I come in.**

I bring in independent critics — each one focused on a different dimension — and they go through your work carefully. Not vibes. Not "looks good to me." Every finding has to cite specific evidence from your artifact. If a critic can't show me the proof, I throw out the finding.

```
$ quorum run --target my-report.md --depth standard

Running Quorum (standard depth, critics: correctness, completeness, security) ...

 ◆ QUORUM VERDICT: PASS_WITH_NOTES
 ──────────────────────────────────
   Issues: 0 HIGH · 2 MEDIUM · 1 LOW/INFO  (3 total)

   1. [MEDIUM] §2.4 claims "all implementations use AES-256" but §5.2
      references a ChaCha20 fallback — contradictory coverage claim
   2. [MEDIUM] Rubric criterion RC-007 (threat model) has no matching
      content in any section
   3. [INFO]   Minor: bibliography entry [14] not cited in text

   Cost: $0.42 (12,840 prompt + 3,201 completion tokens)
```

Now you know. Not because you hoped. Because it was checked.

---

## What Makes Me Different

| The usual approach | What I do instead |
|---|---|
| One model reviews its own output | **Separate critics** that never saw the original prompt |
| "Looks great!" — it wrote it, of course it thinks so | Critics come in cold. **No bias from creation** |
| Vague suggestions you can't act on | **Every finding cites evidence** — excerpts, line numbers, schema checks |
| Same effort for a draft vs. production | **Three depth levels**: quick ($0.15), standard ($0.50), thorough ($1.50+) |
| Reviews one file at a time | **Batch + cross-artifact** — validate a directory with consistency checks between files |
| LLM wastes tokens on obvious problems | **Multi-layer pre-screen** catches issues first — 10 deterministic checks + DevSkim + Ruff + Bandit before LLM critics run |
| Each review starts from zero | **Learning memory** — recurring patterns auto-promote to mandatory checks |

You wouldn't ship code without tests. I'm here so you don't ship AI outputs without validation either.

---

## The Deeper Question

Validation is the beginning. The question that matters in compliance, audits, and anything with stakes is **substantiation**: *Can you prove the output meets a specific standard?*

That's what rubrics are for. A rubric encodes a standard — OWASP ASVS, SOC 2 controls, your internal style guide — as testable, machine-readable criteria. I evaluate evidence against those criteria. You get findings with citations, not feelings.

I ship with three rubrics (research-synthesis, agent-config, python-code). Need one for your domain? → [Rubric Building Guide](docs/guides/RUBRIC_BUILDING_GUIDE.md)

---

## Install

**PyPI** (recommended):
```bash
pip install quorum-validator
quorum run --target your-file.py --depth standard
```

**From source:**
```bash
git clone https://github.com/SharedIntellect/quorum.git
cd quorum/reference-implementation
pip install -e .
```

**OpenClaw / ClawHub:**
```bash
openclaw skills add dacervera/quorum
```

### Platform Ports

Quorum also runs natively inside other agent platforms — no API key required:

| Port | What it does | Install |
|------|-------------|---------|
| **[Copilot CLI](ports/copilot-cli/)** | Full validation as a GitHub Copilot skill — stdlib-only pre-screen + 5 LLM critics | Copy `ports/copilot-cli/` to `~/.copilot/skills/quorum/` |
| **[Claude Code](ports/claude-code/)** | Quorum validation via Claude Code subscription tokens | Copy `ports/claude-code/` to your project |

### Configuration

You'll need a model that can reason well. I auto-detect your provider on first run.

| Tier | Models | Best for |
|------|--------|----------|
| **Recommended** | Claude Opus, GPT-5+ | Thorough depth, cross-artifact consistency |
| **Great** | Claude Sonnet, Gemini 2.0+ | Quick and standard — where most work lives |
| **Functional** | Claude Haiku, GPT-4o | Lighter validation, less nuance on edge cases |

Details: [Model Requirements](docs/getting-started/MODEL_REQUIREMENTS.md) · [Config Reference](docs/configuration/CONFIG_REFERENCE.md)

---

## Documentation

| Getting Started | Guides | Architecture |
|---|---|---|
| [Installation](docs/getting-started/INSTALLATION.md) | [Rubric Building Guide](docs/guides/RUBRIC_BUILDING_GUIDE.md) | [Full Specification](SPEC.md) |
| [Quick Start](docs/getting-started/QUICK_START.md) | [Cross-Artifact Design](docs/guides/CROSS_ARTIFACT_DESIGN.md) | [The Nine Critics](docs/architecture/THE_NINE.md) |
| [For Beginners](docs/getting-started/FOR_BEGINNERS.md) | [Security Critic Framework](docs/critics/SECURITY_CRITIC_FRAMEWORK.md) | [Implementation Notes](docs/architecture/IMPLEMENTATION.md) |
| [Model Requirements](docs/getting-started/MODEL_REQUIREMENTS.md) | [Code Hygiene Framework](docs/critics/CODE_HYGIENE_FRAMEWORK.md) | [Changelog](docs/CHANGELOG.md) |
| [Config Reference](docs/configuration/CONFIG_REFERENCE.md) | [Contributing](CONTRIBUTING.md) | [External Reviews](docs/reviews/EXTERNAL_REVIEWS.md) |

New to AI agent tooling? → [FOR_BEGINNERS.md](docs/getting-started/FOR_BEGINNERS.md) — I'll start from the very beginning.

Full documentation index: [docs/](docs/README.md)

---

<p align="center">
  MIT License · <a href="https://sharedintellect.com">SharedIntellect</a> · 2026
</p>


path: SHIPPING.md
content: # Shipping Checklist

Follow this checklist before merging any feature PR to `main`. Both humans and agents (including Claude Code) must complete all steps.

---

## 1. Code Quality
- [ ] All tests pass (`pytest`)
- [ ] No regressions in existing functionality

## 2. Update Manifest
- [ ] `critic-status.yaml` updated (add/change status for shipped feature)
- [ ] Version string bumped in `critic-status.yaml`
- [ ] Version string bumped in `reference-implementation/pyproject.toml`

## 3. Documentation Sweep
Run `python tools/validate-docs.py` to find stale references, then fix:

- [ ] `CHANGELOG.md` — add entry for this release
- [ ] `README.md` — version badge in header
- [ ] `README.md` — "What I can do today" section accurate
- [ ] `README.md` — critic count in depth table footnote
- [ ] `README.md` — architecture diagram labels (shipped vs roadmap)
- [ ] `README.md` — "What's coming" section (remove shipped items)
- [ ] `SPEC.md` — status matrix / implementation checklist updated
- [ ] `SPEC.md` — shipped critic counts in prose
- [ ] `docs/getting-started/FOR_BEGINNERS.md` — critic counts and cost estimates
- [ ] `docs/configuration/CONFIG_REFERENCE.md` — critic status markers
- [ ] `docs/getting-started/TUTORIAL.md` — critic descriptions and status markers
- [ ] `docs/architecture/IMPLEMENTATION.md` — critic status markers and depth profiles
- [ ] `SKILL.md` — critic counts in depth descriptions

**⚠️ Note on Cross-Consistency:** Cross-Consistency is a shipped critic but requires the `--relationships` flag. When updating counts, use language like: "6 shipped critics (including Cross-Consistency, activated with `--relationships`)".

## 4. Validation
- [ ] `python tools/validate-docs.py` passes (exit 0)
- [ ] `tools/boundary-scan.sh` passes (exit 0)

## 5. Commit & Ship
- [ ] Commit message includes: what shipped, critic N of 9, known limitations
- [ ] Tag release if version bump warrants it

---

## Why This Exists

PR #8 (Tester critic) shipped with 10+ doc files still claiming "4 critics." The Cross-Consistency critic shipped at some point with zero doc updates. This checklist + `validate-docs.py` + CI enforcement exists to prevent that from happening again.


path: SPEC.md
content: # Quorum — Architectural Specification

**Version:** 3.0  
**Last Updated:** March 2026  
**Status:** Documented and partially implemented — see §3 for implementation status per component  
**Platform:** Designed for [OpenClaw](https://openclaw.ai) agent systems. Cross-platform compatibility with other agent frameworks is under active exploration — see [MODEL_REQUIREMENTS.md](docs/getting-started/MODEL_REQUIREMENTS.md) for supported models and platforms.

---

## 1. Overview

Quorum is a quality assurance framework with a nine-agent target architecture. Currently, 6 critics are implemented (Correctness, Completeness, Security, Code Hygiene, Cross-Artifact Consistency†, and Tester), with 3 additional critics planned. †Cross-Consistency is activated with the `--relationships` flag. The framework is designed to rigorously evaluate multi-agent systems, configurations, research, code, and operational procedures against domain-specific rubrics. It combines:

- **Parallel specialized critics** (9 agents with distinct expertise)
- **Grounded evidence requirement** (every critique must cite tool-verified proof)
- **Intelligent delegation** (Tomasev et al., 2026 principles)
- **Learning memory** (persistent failure-pattern tracking; recurring patterns auto-promote to mandatory critic checks)
- **Cost-aware depth control** (three execution profiles for different budgets)

Quorum treats validation as infrastructure, not an afterthought.

---

## 2. Design Principles

### 2.1 Multi-Critic Architecture (Reflexion + Replacing Judges with Juries)

A single model reviewing a long prompt generates:
- Single point of failure (one model's blindspots)
- Hand-waving without evidence (LLMs can justify anything)
- ~~No learning across validations~~ → **Solved in v0.5.3:** Learning memory tracks recurring patterns and promotes high-frequency findings to mandatory checks

Quorum's target architecture has **nine specialized agents** — six critics are currently shipped, with three more planned:

**Shipped critics (6):**

1. **Correctness Critic** — Factual accuracy, logical consistency, claim support
2. **Completeness Critic** — Coverage gaps, missing requirements, unaddressed edge cases
3. **Security Critic** — Vulnerability patterns, permission issues, injection risks
4. **Code Hygiene Critic** — ISO 25010 maintainability, naming, complexity metrics
5. **Cross-Artifact Consistency Critic**† — Inter-file consistency (activated with `--relationships`)
6. **Tester Agent** — Finding verification: deterministic checks + LLM claim validation

**Shipped infrastructure agents:**

7. **Fixer Agent** — Proposes and applies fixes for CRITICAL/HIGH findings (optional, 1-2 loops max)
8. **Aggregator** — Merges findings, resolves conflicts, recalibrates confidence
9. **Supervisor** — Manages workflow, checkpoints, final verdict

**Planned critics (not yet built):**

10. **Architecture Critic** — Design coherence, pattern consistency, scalability concerns
11. **Delegation & Coordination Critic** — Span of control, reversibility, bidirectional contracts
12. **Style Critic** — Writing quality, tone consistency, formatting standards

Critics don't vote. The Aggregator synthesizes their findings into a final verdict with explicit confidence levels.

### 2.2 Grounded Evidence Requirement

Every issue **must** include tool-verified evidence:
- Excerpt from the artifact being reviewed
- Tool output (git log, schema parse, grep result, web search, shell execution)
- Citation to the rubric criterion it violates

The Aggregator **rejects ungrounded claims**. This single constraint is what separates useful critique from LLM hand-waving.

Example:
```json
{
  "issue": "Missing model assignment in worker agent",
  "severity": "CRITICAL",
  "evidence": {
    "tool": "grep",
    "result": "workers:\n  - name: researcher\n    // NO 'model' field",
    "citation": "Rubric § 2.4: Every agent must have explicit model assignment"
  }
}
```

### 2.3 Intelligent Delegation (Tomasev et al., 2026)

Quorum delegates to critics using five principles from "Towards an Intelligent Assessment Framework for Delegation in AI":

1. **Bidirectional Contracts** — Each critic receives explicit acceptance criteria and resource guarantees
2. **Five-Axis Monitoring Profiles** — Target (process/outcome) × Observability (direct/event) × Control (active/passive) × Frequency × Transparency
3. **Dynamic Re-Delegation** — If a critic fails, the Supervisor can downgrade to a cheaper model or escalate
4. **Trust as Runtime Primitive** — Critics earn trust through demonstrated competence; monitoring intensity scales with trust
5. **Reversibility-Aware Decisions** — Configuration changes (reversible) require less scrutiny than deployment decisions (irreversible)

### 2.4 Learning Memory

After each validation run, the system captures new failure patterns in `known_issues.json`:

```json
{
  "ML-001": {
    "pattern": "Missing bidirectional contract in spawned agent",
    "severity": "CRITICAL",
    "frequency": 12,
    "first_seen": "2026-01-15",
    "last_seen": "2026-02-18",
    "source_runs": ["run-001", "run-047", "run-089"],
    "meta_lesson": "Automation opportunity: validate-contracts.sh for mandatory pre-flight checks"
  }
}
```

High-frequency patterns automatically promote to mandatory checks in future runs.

### 2.5 Cost-Aware Depth Control

Three execution profiles balance rigor, speed, and cost:

| Depth | Critics | Fix Loops | Runtime | Use Case |
|-------|---------|-----------|---------|----------|
| **quick** | Correctness, Completeness | 0 | 5-10 min | Fast feedback; low stakes |
| **standard** | + Security | 0 | 15-30 min | Most work; default |
| **thorough** | All shipped critics | 1 (apply + re-verify) | 30-60 min | Critical decisions; production |

Pre-screen (10 built-in checks + optional DevSkim SAST, §3.1) runs before LLM critics at all depth levels. Fix loops are implemented (Phase 1.5): the Fixer proposes text replacements for CRITICAL/HIGH findings, applies them, and re-runs critics to verify resolution. Full 9-critic panels are the roadmap target for thorough depth.

### 2.6 Transparency Over Convenience

> *"In a judgment system, always trade toward transparency over convenience."*

This is a standing design axiom. When implementation choices arise — such as whether to pass a verdict or raw findings to the next stage, whether to surface intermediate outputs, or whether to collapse multi-locus findings into a summary — Quorum always chooses the option that preserves traceability and exposes reasoning.

Concrete consequences:
- **Phase 2 receives findings, not verdicts** from Phase 1. The Cross-Artifact critic sees what was discovered, not a collapsed judgment, so it can reason independently.
- **Pre-screen results are always written** to `prescreen.json`, even when all checks pass.
- **Run directories are immutable** — outputs are written once and never modified in place.
- **Aggregator decisions are logged** — every dedup and conflict resolution is recorded, not just the outcome.

---

## 3. Architecture

### 3.1 Pre-Screen Layer *(implemented)*

Before any LLM critic runs, Quorum executes a two-pass deterministic pre-screen against the target artifact.

**Pass 1: Built-in checks** — 10 rule-based regex/parse checks. No API costs, completes in milliseconds.

| Check ID | What It Catches |
|----------|----------------|
| PS-001 | Hardcoded absolute paths (portability risk) |
| PS-002 | Credential patterns (API keys, tokens, passwords) |
| PS-003 | PII patterns (email addresses, SSNs, phone numbers) |
| PS-004 | JSON syntax errors |
| PS-005 | YAML syntax errors |
| PS-006 | Python syntax errors |
| PS-007 | Broken internal links (file references that don't resolve) |
| PS-008 | TODO/FIXME/HACK markers |
| PS-009 | Trailing whitespace and mixed line endings |
| PS-010 | Empty file or effectively-empty file |

**Pass 2: DevSkim SAST** — When [DevSkim](https://github.com/microsoft/DevSkim) is installed, Quorum runs it as a second linter pass. DevSkim catches security patterns that regex checks miss (weak crypto, insecure defaults, known-vulnerable API usage). Results are merged into the same pre-screen evidence block. DevSkim is optional — if not installed, Pass 1 runs alone with no degradation.

Pre-screen results are written to `prescreen.json` in the run directory. If any check has severity `CRITICAL` or `HIGH`, the artifact may be rejected before LLM critics are invoked, depending on depth profile. Pre-screen findings are passed as context to Phase 1 critics.

### 3.2 The Agents: Implemented vs. Specified

```
Supervisor (Orchestrator)
├─ Correctness Critic (Tier 2)      [IMPLEMENTED]
├─ Completeness Critic (Tier 2)     [IMPLEMENTED]
├─ Security Critic (Tier 1)         [IMPLEMENTED] — grounded: OWASP ASVS 5.0, CWE Top 25, NIST SA-11; see docs/critics/SEC02_BUSINESS_LOGIC_VALIDATION.md for business logic workflow
├─ Code Hygiene Critic (Tier 2)     [IMPLEMENTED] — grounded: ISO 25010:2023, CISQ
├─ Cross-Artifact Consistency       [IMPLEMENTED] — Phase 2, separate from BaseCritic
├─ Architecture Critic (Tier 2)     [SPECIFIED, not yet built]
├─ Delegation Critic (Tier 1)       [SPECIFIED, not yet built]
├─ Style Critic (Tier 2)            [SPECIFIED, not yet built]
├─ Tester (Tier 2, tools: grep/web/exec) [IMPLEMENTED] — L1 deterministic + L2 LLM claim verification; see docs/critics/TESTER_CRITIC_BRIEF.md
├─ Fixer (Tier 1, optional)         [IMPLEMENTED — proposal mode + re-validation loops]
├─ Aggregator (Tier 1)              [IMPLEMENTED]
└─ Supervisor (Tier 1, final)       [IMPLEMENTED]
```

Model assignments reflect Tomasev delegation: judgment-heavy roles (Security, Aggregator, Supervisor) use your strongest model (Tier 1); execution-heavy roles use a capable but cost-efficient model (Tier 2). For example, Tier 1 might be Claude Opus or GPT-4, and Tier 2 might be Claude Sonnet or GPT-4o-mini.

### 3.3 Two-Phase Pipeline

```
quorum run --target file --relationships manifest.yaml
  │
  ├─ Pre-Screen (§3.1) → prescreen.json
  │
  ├─ Phase 1: Supervisor dispatches single-file critics (parallel)
  │   ├─ correctness
  │   ├─ completeness
  │   ├─ security (framework-grounded)
  │   └─ code_hygiene (framework-grounded)
  │
  ├─ Phase 2: Cross-Artifact Consistency (if --relationships provided)
  │   └─ Evaluates declared relationships between files
  │       Receives Phase 1 findings (NOT verdicts) as context
  │
  ├─ Aggregator → dedup, resolve conflicts, assign verdict
  │
  └─ Output: PASS / PASS_WITH_NOTES / REVISE / REJECT
```

**Phase coordination contract:** Phase 1 critics produce `Finding` objects with grounded evidence. The Aggregator does not run between phases. Phase 2 receives the raw Phase 1 findings as a serialized list — it can observe what was found, but no intermediate verdict has been assigned. This is required by Design Axiom §2.6: transparency over convenience.

### 3.4 Cross-Artifact Consistency *(Phase 2)*

When `--relationships` is provided, Quorum loads a YAML manifest declaring relationships between files and dispatches the Cross-Artifact Consistency critic.

**Manifest format:**
```yaml
version: "1.0"
relationships:
  - source: src/api_handler.py
    target: docs/api-spec.md
    type: implements
  - source: docs/api-spec.md
    target: src/api_handler.py
    type: documents
  - source: quorum/critics/security.py
    target: quorum/configs/standard.yaml
    type: delegates
  - source: data/output-schema.json
    target: src/pipeline.py
    type: schema_contract
```

**Relationship types:** `implements` | `documents` | `delegates` | `schema_contract`

**Locus model:** Cross-artifact findings use a `Locus` object with two anchor points:

```json
{
  "loci": [
    {
      "file": "src/api_handler.py",
      "role": "source",
      "excerpt": "def get_user(id): ...",
      "source_hash": "sha256:abc123"
    },
    {
      "file": "docs/api-spec.md",
      "role": "target",
      "excerpt": "GET /users/{id} — returns user object",
      "source_hash": "sha256:def456"
    }
  ],
  "relationship_type": "implements",
  "issue": "Handler returns 200 on missing user; spec declares 404",
  "severity": "HIGH"
}
```

`source_hash` pins each finding to the artifact version that was evaluated, enabling reproducibility audits.

**Path safety:** All paths in the manifest are resolved and boundary-checked against the project root. Traversal outside the declared boundary raises an error before evaluation begins.

### 3.5 The Workflow

1. **Intake** — Supervisor receives the artifact (config, research, code), target rubric, and optional relationships manifest
2. **Pre-Screen** — built-in checks + DevSkim SAST run (§3.1); results written to `prescreen.json`
3. **Phase 1 Dispatch** — Supervisor provides each critic with:
   - The artifact excerpt relevant to their domain
   - The rubric criteria they must evaluate
   - Required evidence format
   - Pre-screen results as context
4. **Phase 1 Parallel Execution** — Shipped critics run in parallel (see §3.2)
5. **Phase 2 Dispatch** (if `--relationships`) — Cross-Artifact critic receives:
   - Both artifacts for each declared relationship
   - Phase 1 findings (not verdicts) as context
6. **Aggregation** — Aggregator:
   - Deduplicates issues across all phases and critics
   - Resolves conflicts (if critics disagree, escalates to Supervisor)
   - Recalibrates confidence scores
7. **Verdict** — Supervisor assigns final verdict:
   - **PASS** — No issues or only LOW-severity findings
   - **PASS_WITH_NOTES** — Issues found, all addressable, recommendations provided
   - **REVISE** — HIGH/CRITICAL issues require rework; Supervisor provides guidance
   - **REJECT** — Unfixable architectural problems; restart required
8. **Learning** — System extracts and logs new failure patterns *(implemented in v0.5.3)*

### 3.6 File-Based Artifact Passing

All communication between agents uses file-based artifacts, not in-memory variables:

```
run-manifest.json                   ← Supervisor's execution plan
artifact.yaml                       ← What's being validated
rubric.json                         ← Validation criteria
prescreen.json                      ← Deterministic pre-screen results (PS-001–PS-010)
critics/
├── correctness-findings.json
├── completeness-findings.json
├── security-findings.json
├── code_hygiene-findings.json
└── cross_consistency-findings.json  ← Phase 2 (if --relationships used)
aggregator-synthesis.json           ← Merged findings
known_issues.json                   ← Learning memory (updated)
verdict.json                        ← Final result
report.md                           ← Human-readable summary
```

This enforces:
- Determinism (tool output is reproducible)
- Auditability (every change is logged to a file)
- Parallelism (critics don't block each other)
- Safety (no in-memory prompt injection vectors)

---

## 4. Rubric System

Rubrics define what "good" looks like for a specific artifact type. They're JSON documents with:

```json
{
  "name": "Swarm Configuration Rubric",
  "domain": "multi-agent-systems",
  "version": "2.0",
  "criteria": [
    {
      "id": "CRIT-001",
      "criterion": "Every agent has explicit model assignment",
      "severity": "CRITICAL",
      "evidence_required": "grep output showing 'model: ...' in CONFIG",
      "why": "Without model assignment, the system uses defaults unpredictably"
    },
    {
      "id": "CRIT-002",
      "criterion": "Bidirectional contracts exist for all delegations",
      "severity": "CRITICAL",
      "evidence_required": "Schema parse of contract section showing delegator+delegatee commitments",
      "why": "Tomasev delegation principle: both sides must be protected"
    },
    // ... more criteria
  ]
}
```

Rubrics are:
- **Domain-specific** (research synthesis ≠ code review ≠ config audit)
- **Composable** (build custom rubrics by mixing/extending standard ones)
- **Versionable** (rubrics evolve; track changes)
- **Machine-readable** (Supervisor validates rubric itself before use)

---

## 5. The Learning System

`known_issues.json` is Quorum's "experience memory." After each run:

```json
{
  "ML-001": {
    "pattern": "Missing I/O contracts in spawned agent",
    "severity": "CRITICAL",
    "source_papers": ["Tomasev et al. 2026 § 4.3"],
    "frequency": 12,
    "first_seen": "2026-01-15",
    "last_seen": "2026-02-18",
    "source_runs": ["run-001", "run-047"],
    "automation_opportunity": "Pre-flight tool: validate-contracts.sh checks all agent spawns"
  }
}
```

Rules:
- Patterns with frequency ≥ 10 become **mandatory checks** in future validation runs
- Patterns with frequency ≥ 5 trigger **automation opportunities** (design tools to check this deterministically)
- Patterns go stale after 60 days without recurrence (removed from mandatory list)

This learning architecture is implemented as of v0.5.3. See `quorum issues list|promote|reset` CLI commands and `--no-learning` flag.

---

## 6. Trust & Monitoring

Critics earn trust through demonstrated competence:

```
NEW (1st run)
  → PROBATIONARY (2-4 runs, 70%+ accuracy)
    → ESTABLISHED (5-9 runs, 85%+ accuracy)
      → TRUSTED (10+ runs, 95%+ accuracy)
```

Trust level modifies:
- **Monitoring intensity** — NEW critics get tighter scrutiny (more re-validation)
- **Approval thresholds** — TRUSTED critics can auto-approve LOW findings
- **Resource allocation** — ESTABLISHED/TRUSTED critics get higher token budgets

This follows Tomasev's "trust as runtime primitive" principle.

---

## 7. Cost Model

| Component | Cost | Amortization |
|-----------|------|--------------|
| Per-run setup (Supervisor intake) | $0.02 | 1 run |
| 6 shipped critics (parallel, max 30min; 9 at full build-out) | $0.15-0.45 | 1 run |
| Aggregator synthesis | $0.01 | 1 run |
| Tester tools (grep, git, web, exec) | $0.00 | amortized |
| Learning update (`known_issues.json`) | $0.00 | amortized |
| **Total per run (standard depth)** | **~$0.20-0.50** | **1 run** |

Additional runs on related artifacts reuse critic prompts and tools, amortizing costs further.

---

## 8. Implementation Checklist

Status as of v0.7.3 (reference implementation):

- [x] LLM provider — LiteLLM universal provider (100+ models, any tier combination)
- [x] File-based artifact passing (no in-memory state between agents)
- [x] Pre-screen layer — 10 built-in checks (PS-001–PS-010) + DevSkim SAST integration
- [x] 6 critics implemented — Correctness, Completeness, Security, Code Hygiene, Cross-Artifact Consistency† (`--relationships`), Tester (L1 + L2)
- [x] Rubric system (JSON schema + validator, 3 built-in rubrics: research-synthesis, agent-config, python-code)
- [x] Batch/multi-file validation with `BatchVerdict`
- [x] Aggregator synthesis logic (conflict resolution)
- [x] Verdict assignment logic (PASS/PASS_WITH_NOTES/REVISE/REJECT)
- [x] Depth preset system (quick/standard/thorough YAML configs)
- [x] Path traversal security (boundary enforcement)
- [x] Exit codes (0/1/2)
- [x] Parallel critic dispatch (ThreadPoolExecutor, max 6 critics; batch files max 3)
- [x] Fixer agent — proposal mode + re-validation loops (Phase 1.5; proposes and applies text replacements for CRITICAL/HIGH, then re-runs critics to verify)
- [x] Python code rubric (25 criteria, PC-001–PC-025, auto-detects on .py files)
- [x] Learning memory system (known_issues.json frequency tracking + mandatory check promotion — shipped in v0.5.3)
- [ ] Remaining 3 critics — Architecture, Delegation, Style
- [ ] Trust/monitoring system (per-critic accuracy tracking)

See IMPLEMENTATION.md for a reference walkthrough.

---

## 9. Theoretical Grounding

Quorum is built on these peer-reviewed papers:

| Paper | Contribution |
|-------|--------------|
| Shinn et al. (2023), Reflexion | Iterative self-critique, learning from failures |
| Verga et al. (2024), Replacing Judges with Juries | Multi-critic consensus, conflict resolution |
| Cai et al. (2024), LATM | Tool-making paradigm, deterministic execution |
| Wölflein et al. (2025), ToolMaker | Closed-loop tool generation, autonomous debugging |
| Tomasev et al. (2026), Intelligent AI Delegation | Bidirectional contracts, monitoring profiles, trust primitives |

---

## 10. Known Limitations & Roadmap

### Current Limitations (v3.0)

- Only **6 of 9 critics are implemented** (Architecture, Delegation, Style are specified but not built)
- Rubric panel is **static** (doesn't specialize per artifact type dynamically)
- **No critic-to-critic debate** (relies on Aggregator to resolve conflicts)
- Learning is **frequency-based** only (no semantic deduplication of patterns yet)
- **Confidence calibration** replaced by criteria coverage counts in v0.6.1
- **Trust/monitoring system** is not yet implemented

### Planned

- Remaining critics: Architecture, Delegation, Style
- ~~Re-validation loops~~ → **Shipped in v0.5.3**
- ~~Learning memory~~ → **Shipped in v0.5.3**
- Dynamic critic specialization (spawn domain-specific critics on-demand)
- Critic debate mode (when two critics conflict, run a structured debate)
- Semantic pattern deduplication (group similar issues under one ML pattern)
- Empirical confidence calibration (long-term tracking of verdict accuracy)
- Domain-specific rubric packs (compliance, security, infrastructure)

---

## 11. Getting Started

1. Read [IMPLEMENTATION.md](docs/architecture/IMPLEMENTATION.md) for a reference walkthrough
2. Review [examples/](examples/) for your use case
3. Adapt built-in rubrics or build custom ones (see [RUBRIC_BUILDING_GUIDE.md](docs/guides/RUBRIC_BUILDING_GUIDE.md))
4. Run the tutorial: `quorum run --target examples/sample-research.md --depth quick`

---

**Quorum is a production-oriented, early-stage validation framework.**  
*Architecture is sound and tested, with 6 critics shipped (all callable), learning memory, fix re-validation, batch processing, and the Tester verification layer all production-ready. Trust and monitoring systems are specified but not yet wired.*


---

> ⚖️ **LICENSE** — Not part of the operational specification above.
> This file is part of [Quorum](https://github.com/SharedIntellect/quorum).
> Copyright 2026 SharedIntellect. MIT License.
> See [LICENSE](https://github.com/SharedIntellect/quorum/blob/main/LICENSE) for full terms.


path: branding/BRAND_GUIDE.md
content: # Quorum — Brand Guide

## Name

**Quorum**  
Formal: *Quorum*  
Abbreviated: *Quorum*  
Never: *Validator Swarm*, *TheQuorum*, *Q9*

---

## Core Identity

Quorum is **infrastructure**, not a product or a research project.  
Its tone is: **precise, confident, no-nonsense, evidence-driven.**

What it's not:
- Clever or cute (it's a quality gate, not a chatbot)
- Academic (it ships and runs in production)
- Defensive (it's proud of its gaps and roadmap)

The one-line: **"A production-grade quality gate for agent systems."**

---

## Logo

### Primary Mark: The Q Wordmark

The Quorum logo is a stylized **"Quorum"** wordmark with an oversized capital Q. Flowing, interweaving gradient lines emerge from the Q's form, evoking **convergence, consensus, and interconnection** — the core concept of agents reaching quorum. The lines terminate in small circular nodes representing connection endpoints.

**Key files:**
- `quorum-logo-final.jpg` — Primary logo (white background)
- `quorum-readme-banner.png` — README header with tagline
- `quorum-social-preview-final.png` — GitHub/social preview (1280×640)

**Usage rules:**
- Minimum clear space: 1× the height of the lowercase "o" on all sides
- Minimum size: 120px wide for digital, 1" wide for print
- Do not rotate, stretch, recolor, or add effects
- On dark backgrounds, use a version with adjusted contrast (TBD)

### Color Palette

| Name | Hex | Usage |
|------|-----|-------|
| **Quorum Navy** | `#1a2744` | Primary text, wordmark |
| **Quorum Cyan** | `#2ba4c8` | Primary accent, taglines, badges, flowing lines |
| **Mid Blue** | `#2078a8` | Secondary accent, rating badges |
| **Light Aqua** | `#7dd4e8` | Gradient start on flowing lines |
| **Slate** | `#555555` | Secondary text, feature descriptions |
| **White** | `#FFFFFF` | Primary background |
| **Caution Amber** | `#FFB300` | PASS_WITH_NOTES, warnings, partial states |
| **Critical Red** | `#FF1744` | CRITICAL issues, REJECT verdict |

**Rationale:** The navy + cyan palette communicates professionalism and trust while the gradient lines add dynamism. The issue severity colors (green/amber/red) map directly to Quorum's verdict taxonomy.

### Typography

- **Logo wordmark:** Bold sans-serif (as rendered in logo asset)
- **Headlines:** Exo 2 Bold (matches SharedIntellect site)
- **Body:** Exo 2 Regular or Inter Regular
- **Code:** JetBrains Mono or Fira Code
- **Avoid:** Serif fonts, decorative fonts, anything that reads as "startup-y"

---

## Taglines

**Primary:** "A production-grade quality gate for agent systems."

**Alternatives:**
- "Nine critics. One verdict. Zero hand-waving."
- "Evidence-first validation for multi-agent systems."
- "What your agent system ships shouldn't surprise you."
- "Build agents that hold up. Run Quorum."

---

## Social / Community

- **X/Twitter:** Share under **#Quorum**
- **GitHub:** `SharedIntellect/quorum` — Discussions + Issues for community feedback
- **Authors:** Daniel Cervera and Akkari

---

## What Not To Do

- Don't use corporate stock imagery
- Don't use the "AI brain" or neural network aesthetic
- Don't call it a "framework" (it's a system)
- Don't soften the verdict language ("areas for improvement" → "CRITICAL issues")
- Don't imply it's a chatbot or conversational interface
- Don't place the logo on busy/patterned backgrounds
- Don't use the old green/black color scheme (deprecated v1–v4 concepts)

---

*Branding should communicate what the product actually is: rigorous, production-grade, evidence-driven infrastructure.*


path: critic-status.yaml
content: # critic-status.yaml — Single source of truth for shipped features
# Used by tools/validate-docs.py to detect stale documentation.
# Update this file FIRST when shipping a critic or feature.

version: "0.7.3"
spec_version: "3.0"

critics:
  correctness:
    status: shipped
    callable: true
    since: "0.3.0"
    description: "Factual accuracy, citation validity, logical consistency"
  completeness:
    status: shipped
    callable: true
    since: "0.3.0"
    description: "Coverage gaps, missing sections, rubric criterion coverage"
  security:
    status: shipped
    callable: true
    since: "0.4.0"
    description: "OWASP ASVS 5.0, CWE Top 25, NIST SA-11, business logic"
  code_hygiene:
    status: shipped
    callable: true
    since: "0.4.0"
    description: "ISO 25010:2023, CISQ quality measures, maintainability"
  cross_consistency:
    status: shipped
    callable: true
    since: "0.5.0"
    description: "Inter-file consistency via relationships manifest (requires --relationships flag)"
    note: "Activated with --relationships flag; not part of default depth panel"
  tester:
    status: shipped
    callable: true
    since: "0.6.0"
    description: "Finding verification — L1 deterministic + L2 LLM claim checking"
    note: "Phase 3 in pipeline. L1 at standard depth, L1+L2 at thorough. Quick: skipped."
    depth_configs:
      - standard
      - thorough
  architecture:
    status: planned
    callable: false
    description: "Structural design, modularity, dependency analysis"
  delegation:
    status: planned
    callable: false
    description: "Multi-agent coordination, responsibility boundaries"
  style:
    status: planned
    callable: false
    description: "Writing quality, tone consistency, formatting standards"

# What critics actually run at each depth level via `quorum run`
depth_configs:
  quick:
    - correctness
    - completeness
  standard:
    - correctness
    - completeness
    - security
  thorough:
    - correctness
    - completeness
    - security
    - code_hygiene

features:
  prescreen:
    status: shipped
    description: "10 deterministic checks + DevSkim + Ruff + Bandit + PSScriptAnalyzer"
  fix_loops:
    status: shipped
    description: "Re-validation loops with fixer proposals, up to 3 iterations"
  learning_memory:
    status: shipped
    description: "Recurring pattern tracking via known_issues.json"
  batch_validation:
    status: shipped
    description: "Directory/pattern validation with crash-resilient progressive saves"
  cost_tracking:
    status: shipped
    description: "Per-call token counts, cost estimates, budget caps"
  audit_reports:
    status: shipped
    description: "CSV audit detail + summary reports with SHA-256 integrity"
  time_estimation:
    status: shipped
    description: "Pre-run time + cost estimates with --estimate-time flag"


path: docs/CHANGELOG.md
content: # Changelog

All notable changes to Quorum will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [0.7.3] — 2026-03-15

### Fixed — Pre-Existing Code Findings Cleanup

- **`__version__` corrected** — `reference-implementation/quorum/__init__.py` now reads `0.7.3` (was stuck at `0.5.3` since initial release; PyPI showed `0.7.2` but runtime `__version__` did not match)
- **Broad exception handling narrowed** (prescreen.py) — PS-006 `except Exception` narrowed to `(OSError, UnicodeDecodeError)`; external tool integrations (Ruff, DevSkim, Bandit, PSSA) now log with `exc_info=True` for traceback capture on unexpected failures
- **PSScriptAnalyzer security documentation** (prescreen.py) — Added comprehensive `# SECURITY:` comment explaining the PowerShell single-quote injection trust model for `_run_pssa`
- **SPEC.md §2.1 overclaiming fixed** — Numbered agent list now separates shipped critics (6) from planned critics (3) and infrastructure agents; Code Hygiene and Cross-Artifact Consistency added (were missing from list despite being shipped)
- **validate-docs.py hardcoded range removed** — `check_hardcoded_counts` `range(2, 9)` replaced with parameterized bounds derived from manifest total critic count
- **validate-docs.py SRP cleanup** — Extracted `_extract_pyproject_version()` and `_extract_badge_versions()` as pure functions; `main()` refactored to delegate to `validate_docs()` (eliminated duplicated validation loop)
- **Test exception handling** (test_batch_resilience.py) — Broad `except Exception` narrowed to `(json.JSONDecodeError, OSError)` with explanatory comment

---

## [0.7.2] — 2026-03-12

### Documentation — Stale Content Cleanup (PR #18)

- **External reviews removed**: 3 frontier model review files (GPT-5.4, Opus 4.6, Grok 4.20) and ideas-backlog.md deleted — reviews were run against v0.5.3 and no longer reflect current capabilities. `EXTERNAL_REVIEWS.md` replaced with community invitation template.
- **13 stale content findings resolved** across SPEC.md, IMPLEMENTATION.md, CONFIG_REFERENCE.md, TUTORIAL.md, SKILL.md, RUBRIC_BUILDING_GUIDE.md, SECURITY_CRITIC_FRAMEWORK.md, CODE_HYGIENE_FRAMEWORK.md, TESTER_CRITIC_BRIEF.md, and INSTALLATION.md:
  - Tester marked as shipped (was "SPECIFIED, not yet built")
  - Critic counts updated to 6 shipped across all docs
  - Learning memory marked as shipped (v0.5.3) across all docs
  - Confidence scores replaced with coverage language (v0.6.1)
  - Version headers updated to v0.7.2
  - Depth profile critic counts aligned with critic-status.yaml
  - Broken cross-reference paths fixed (TESTER_CRITIC_BRIEF, INSTALLATION)
  - Security and Code Hygiene framework status checklists updated

### Documentation — Docs Restructure & README Rewrite (PR #16)

- **README.md** rewritten: ~400 lines → ~120 lines. Focused on what Quorum is, how to run it, and where to go next. First-person voice preserved.
- **23 docs reorganized** into 6 categorized subdirectories under `docs/`:
  - `getting-started/` — QUICK_START, INSTALLATION, TUTORIAL, FOR_BEGINNERS, MODEL_REQUIREMENTS
  - `architecture/` — IMPLEMENTATION, THE_NINE
  - `critics/` — all critic-specific docs and frameworks
  - `guides/` — RUBRIC_BUILDING_GUIDE, CROSS_ARTIFACT_DESIGN
  - `reviews/` — community and practitioner evaluations
  - `configuration/` — CONFIG_REFERENCE
- **New files:** `docs/getting-started/QUICK_START.md` (zero to running in <5 min), `docs/README.md` (navigation hub)
- All cross-references updated across 30+ files
- `validate-docs.py` exclusion updated to match new `docs/reviews/` path

### Fixed
- 7 broken relative links caused by docs moving one directory deeper (CONFIG_REFERENCE, SPEC, CONTRIBUTING, reference-implementation paths)
- `SKILL.md` frontmatter: "4 independent AI critics" → "6" (matched shipped state)
- `README.md` version badge: v0.7.0 → v0.7.2

---

## [0.7.1] — 2026-03-12

### Fixed — Self-Validation Findings (PR #14)

Ran Quorum self-validation (standard depth) against all shipped artifacts in v0.6.1, Copilot CLI Port, Golden Test Set, and core engine. Fixed **64 findings** across **10 files** — 3 CRITICAL, 16 HIGH, 45 MEDIUM/LOW.

#### Critical Fixes
- **aggregator.py** — INFO-only findings now correctly produce PASS (not PASS_WITH_NOTES), honoring the documented verdict contract
- **pipeline.py** — `resume_batch_validation` now restores original config + relationships_path from manifest (was hardcoding None, causing resumed batches to produce different results than original runs)
- **models.py** — `Locus` now validates `start_line <= end_line` and `_hash_byte_range` raises ValueError on out-of-bounds ranges instead of silently hashing empty bytes

#### High-Impact Fixes
- **Severity normalization** (base.py, cross_consistency.py) — malformed LLM severity strings no longer discard ALL findings for that critic
- **Format-string injection** (cross_consistency.py) — user YAML `scope` field now brace-escaped before `str.format()`
- **Deterministic dedup** (aggregator.py) — findings sorted by severity before greedy matching; output no longer depends on input ordering
- **CancelledError handling** (pipeline.py) — cancelled futures in batch loops no longer crash pipeline
- **Library-safe prescreen** (quorum-prescreen.py) — `run_prescreen()` returns error dicts instead of calling `sys.exit()`

#### Tests
- 879 passed, 0 failures
- 2 tests updated: `test_info_finding_pass_with_notes` and `test_verdict_json_serialization_never_none` (codified buggy behavior; now corrected)

#### Process
- Added mandatory self-validation step to Claude Code task template (`~/.claude/tasks/README.md`)

---

## [0.7.0-port.1] — 2026-03-12

### Added
- **Copilot CLI port** at `ports/copilot-cli/` — Quorum's multi-critic validation framework packaged as a GitHub Copilot CLI skill. Zero infrastructure, zero pip dependencies.
  - `SKILL.md` (414 lines): Orchestrator skill — artifact classification, rubric selection, prescreen gating, critic dispatch, verdict aggregation, report generation.
  - `quorum-prescreen.py` (649 lines): Stdlib-only pre-screen implementing PS-001 through PS-010. PyYAML optional for YAML artifact support.
  - 5 critic `.agent.md` files: Verbatim prompts for correctness, completeness, security, code hygiene, and cross-consistency critics.
  - 3 rubric JSON files: Byte-identical copies from reference implementation (python-code, research-synthesis, agent-config).
  - `README.md`: Install and usage guide.
  - `ARCHITECTURE.md`: Port design rationale and mapping to reference implementation.

### Changed
- `.gitignore`: Removed `ports/` exclusion — ports are now tracked for public release.

## [0.7.0] — 2026-03-12

### Added
- **TesterCritic wired into pipeline as Phase 3** (DEC-020). The Tester verifies other critics' findings after Phase 1 (single-file critics) and Phase 2 (cross-consistency), before the Aggregator produces the verdict.
  - **L1 contradictions auto-excluded**: If a critic cites a file/line that doesn't exist or content that doesn't match, the finding is silently removed from verdict calculation. Preserved in tester output for audit.
  - **L2 contradictions annotated**: If the Tester's LLM judges a claim unsupported by evidence, the finding gets a `[L2-CONTRADICTED]` annotation but remains in the verdict. Human decides.
  - **Depth-based activation**: Quick = skip, Standard = L1 only (zero API cost), Thorough = L1 + L2.
- **Tester stats in run manifest**: `verified`, `unverified`, `contradicted`, `verification_rate`, `runtime_ms`.
- **Verification section in report.md** when tester results are present: table of verified/unverified/contradicted counts with per-finding exclusion details.
- 12 new integration tests (`test_tester_pipeline_integration.py`). Total: 879 passed, 6 skipped.

### Changed
- `AggregatorAgent.run()` accepts optional `tester_result` parameter. Applies DEC-020 policy before deduplication.
- `AggregatedReport` model gains `tester_result` and `l1_excluded_count` fields.
- Verdict reasoning now includes excluded finding count when tester ran.
- Removed unnecessary `asyncio.CancelledError` isinstance guards (CancelledError is BaseException in Python 3.9+, never caught by `except Exception`).

## [0.6.1] — 2026-03-11

### Changed
- **Confidence scores replaced with criteria coverage.** Verdicts now report "Coverage: N% of criteria evaluated" with per-critic criteria counts (`criteria_evaluated` / `criteria_total`) instead of fabricated confidence percentages. The previous formula (`0.5 + ratio × 0.45`) always returned ~0.95 since evidence non-emptiness was already enforced upstream — the number was decorative. Coverage counts are honest and verifiable.
- **Aggregator confidence** now computes total criteria evaluated across all critics rather than averaging per-critic scores with skip penalties and agreement bonuses.
- **Report output** shows "N/M criteria" per file and "Coverage: N% of criteria evaluated" in batch and single-file reports. Fixer proposal confidence (LLM self-assessed) is unchanged — it's a legitimately different concept.
- `CriticResult` model gains `criteria_total` and `criteria_evaluated` fields.

### Fixed
- **SPEC.md** status summary updated to reflect 6 shipped critics (was still showing "4/9" from v0.5.x).
- **IMPLEMENTATION.md** status header updated to v0.6.0 with correct critic count and pointer to `critic-status.yaml` as authoritative source.

### Tooling
- **`validate-docs.py` enhanced**: version consistency checks across all doc files, depth config validation against `critic-status.yaml`, fraction pattern detection (DEC-019: no exposed denominators for critic counts), callable vs shipped distinction.
- **`critic-status.yaml` extended**: `callable` flag (shipped vs invocable in pipeline), `spec_version`, `depth_configs` per depth profile.

## [0.6.0] — 2026-03-10

### Added

#### Tester Critic (Milestone #10) — Critic 6 of 9
- **L1 Deterministic Verification**: checks finding loci exist, line numbers are in range, excerpts match source text, file paths are valid
- **L2 LLM Claim Verification**: language model evaluates whether each finding's claim is substantiated by the cited evidence
- 805 lines of code, 994 lines of tests, 76/76 passing
- Merged via PR #8

#### Shipping Process Infrastructure
- **`critic-status.yaml`** — single source of truth for shipped critic and feature status. All doc references derive from this manifest instead of hardcoding counts
- **`tools/validate-docs.py`** — automated doc validation that reads the manifest, scans all public markdown files for stale critic counts, outdated status markers (🔜/Planned for shipped critics), and roadmap references to shipped features. Exit 0 = clean, exit 1 = findings
- **`SHIPPING.md`** — shipping checklist for humans and agents (including Claude Code) to follow before merging feature PRs to main
- **`.github/workflows/validate-docs.yml`** — CI enforcement: runs `validate-docs.py` on every PR to main. Manual checks have repeatedly failed to catch doc staleness; CI is the reliable gate

#### SEC-07: Self-Validation Convergence Problem
- Documented the iterative self-validation convergence problem observed during PR #8
- Root cause: fresh context per run, fix-induced findings, severity recalibration, no delta awareness
- Five mitigations proposed; interim decision: advisory-only self-validation (CRITICAL-only blocking)
- Self-validation moved to non-required in GitHub branch ruleset

### Changed
- **Cross-Consistency critic** now counted as shipped (critic 5 of 9) in all documentation, with visible note that it requires `--relationships` flag
- Version bump to 0.6.0 across `pyproject.toml`, `critic-status.yaml`, and all docs

### Documentation
- **Full doc sweep**: 10 files updated to reflect 6 shipped critics (was inconsistently 4 or 5 across different files)
- README.md architecture diagram updated: Cross-Consistency and Tester moved to "shipped" row; Style remains in "roadmap"
- SPEC.md status matrix, limitations, and roadmap sections updated (6/9 implemented, 3 remaining)
- CONFIG_REFERENCE.md, TUTORIAL.md, IMPLEMENTATION.md — Tester status markers updated from 🔜 to ✅
- FOR_BEGINNERS.md — critic counts and cost estimates updated
- MODEL_REQUIREMENTS.md — cost estimates updated for 6-critic standard depth
- CROSS_ARTIFACT_DESIGN.md — `threat_context` relationship type marked as shipped
- SEC03_EVIDENCE_INTEGRITY.md — Tester status updated from "planned" to "shipped"
- docs/TESTER_CRITIC_BRIEF.md — design brief for Tester critic

### Roadmap Status
- [x] Tester critic (Milestone #10) — L1 deterministic + L2 LLM verification
- [x] Shipping process infrastructure — manifest, validation script, CI gate, checklist
- [ ] Architecture critic (Milestone #9)
- [ ] Confidence calibration (Milestone #6b)
- [ ] Delegation critic (Milestone #11)
- [ ] Style critic (Milestone #12)
- [ ] **Hardware-backed result protection** (Milestone #18)

---

## [0.5.3] — 2026-03-09

### Added

#### Re-Validation Loops (Milestone #5c)
- **Fix-verify cycle**: Fixer proposes text replacements for CRITICAL/HIGH findings → applies them to a working copy → re-runs only the critics that flagged the originals → compares before/after
- **Delta tracking**: each loop reports `improved`, `unchanged`, or `regressed` with finding counts
- **Loop control**: stops early when all findings clear, no proposals generated, or `max_fix_loops` reached (max 3)
- **CLI flag**: `--fix-loops N` overrides depth profile default; `--depth thorough` defaults to 1 loop
- **Immutable originals**: fixed artifacts saved as `artifact-fixed.txt` in run directory; original file never modified
- Per-loop audit trail: `fix-proposals-loop-N.json` saved for each iteration

#### Learning Memory (Milestone #7)
- **Recurring pattern tracking**: `known_issues.json` stores failure patterns with stable SHA-256 pattern IDs, frequency counts, first/last seen dates
- **Auto-promotion**: patterns seen ≥3 times automatically promoted to mandatory checks
- **Critic injection**: mandatory patterns prepended to critic system prompts as natural language instructions — critics are aware of known recurring issues
- **CLI subcommands**: `quorum issues list` (table view), `quorum issues promote --threshold N`, `quorum issues reset`
- **Run integration**: learning memory loads at start, updates after verdict, stats written to `run-manifest.json`
- **Opt-out**: `--no-learning` flag disables for a run
- Atomic file writes (tmp + rename) to prevent corruption

#### Pre-Screen Expansion (Milestone #15)
- **Ruff S-rules**: Python security linting via `ruff check --select S` (S1xx → MEDIUM, S2xx+ → HIGH)
- **Bandit**: Python security analysis with native severity/confidence mapping; deduplicated with Ruff when both present
- **PSScriptAnalyzer**: PowerShell security rules via `pwsh` — filters to 8 security-relevant rules (AvoidUsingInvokeExpression, AvoidUsingBrokenHashAlgorithms, etc.)
- All three tools follow the DevSkim pattern: detect if installed, run if available, graceful degradation if not
- Pre-screen now has 5 layers: built-in regex (PS-001–010) → DevSkim → Ruff → Bandit → PSScriptAnalyzer

#### Crash-Resilient Batch Validation
- **Progressive manifest**: `batch-manifest.json` updates atomically after each file completes (not just at the end). Tracks `completed_files`, `failed_files`, `status` (running/completed/interrupted)
- **Resume**: `quorum run --resume <batch-dir>` reads manifest, skips completed files, re-validates crashed entries, only pays for remaining work
- **Graceful shutdown**: SIGTERM/SIGINT handler saves all in-flight work with `status='interrupted'` before exiting
- **Progressive report**: `batch-report.md` grows as files complete instead of one big write at the end

#### Cost Tracking and Estimation (Milestone #16)
- **Per-call tracking**: CostTracker records prompt/completion tokens and USD cost for every LLM call via `litellm.completion_cost()`
- **Budget cap**: `--max-cost $10` stops gracefully when cumulative cost exceeds threshold — saves all work collected so far
- **Pre-run estimate**: prints "Estimated cost: $X.XX (Y critic calls across Z files). Proceed? [Y/n]" — auto-skips under $0.50 or with `--yes`
- **Cost report**: CLI output shows per-file cost breakdown; `run-manifest.json` includes full cost summary
- Hardcoded rates for Claude, GPT, Gemini, Mistral families; graceful fallback for unknown models

#### CSV Audit Reports (Milestone #17)
- **`audit-detail.csv`**: one row per file — SHA-256 hash, file size, token estimate, analysis start/end/duration, input/output tokens, tokens/sec, models used, verdict, finding count, cost
- **`audit-summary.csv`**: single aggregate row — totals, avg/median tokens/sec, cost by model (JSON), pass/fail counts
- `--audit-report` flag, auto-enabled at `--depth thorough`
- SHA-256 artifact hashes stored in `run-manifest.json`

### Fixed
- **PSScriptAnalyzer command injection**: single quotes in file paths now escaped before passing to `pwsh -Command`
- **Learning memory save resilience**: `save()` wrapped in try/except to prevent crash on write failure

### Documentation
- **docs/README.md** — new documentation index with categorized links to all framework docs
- **docs/SEC02_BUSINESS_LOGIC_VALIDATION.md** — severity calibration table, shared manifest example, Option C success/failure criteria, depth-level mapping (per Devola's review)
- **SPEC.md** — Security Critic links to SEC-02 workflow; Fixer updated to reflect re-validation loops; learning memory sections updated from "planned" to "shipped"
- **README.md** — updated to v0.5.3: all new features, CLI flags, cost tracking, audit reports, crash resilience

### Roadmap Status
- [x] Re-validation loops (Milestone #5c)
- [x] Learning memory (Milestone #7)
- [x] Pre-screen expansion — Ruff, Bandit, PSScriptAnalyzer (Milestone #15)
- [x] PyPI publish (Milestone #14) — `pip install quorum-validator`
- [x] **Cost tracking and estimation** (Milestone #16) — per-call token counts via LiteLLM, cumulative run cost in manifest, pre-run estimate, `--max-cost` budget cap, cost report in CLI output
- [x] **CSV audit reports** (Milestone #17) — per-file SHA-256, timing, token counts, cost breakdown; aggregate summary with median tokens/sec and cost-by-model
- [x] **Crash-resilient batch validation** — progressive manifest saves, `--resume`, graceful SIGTERM/SIGINT shutdown
- [ ] Architecture critic (Milestone #9)
- [ ] Tester critic (Milestone #10)
- [ ] Confidence calibration (Milestone #6b)
- [ ] Delegation critic (Milestone #11)
- [ ] Style critic (Milestone #12)
- [x] Self-validation graduation (GRAD) — CI gate on PRs to main, standard depth, changed files only, $5 budget cap
- [ ] **Hardware-backed result protection** (Milestone #18) — YubiKey KEK wrapping per-run DEKs, encrypted-at-rest validation results, HMAC tamper evidence over run manifests. For sensitive security assessments where findings themselves are high-value targets. CKMS-aligned (SP 800-130/152 principles applied to the validation pipeline)

---

## [0.5.2] — 2026-03-08

### Added

#### DevSkim Pre-Screen Integration
- **New SAST pass: DevSkim** — second security linter alongside the existing 10 deterministic pre-screen checks (~690 lines)
- Catches patterns that regex-based PS-002 misses; runs as part of the pre-screen phase before LLM critics
- Pre-screen layer now has two passes: deterministic rules (PS-001–PS-010) + DevSkim SAST

#### Structured Output Enforcement
- **New utility: `extract_json_from_response()`** — strips markdown fence wrappers (` ```json `) from critic JSON responses before parsing
- **25 tests** covering fence-wrapped, bare JSON, and malformed inputs
- Prevents critic JSON parse failures when LLM wraps output in markdown blocks

#### threat_context Relationship Type
- **New relationship type: `threat_context`** for `quorum-relationships.yaml` — surfaces threat model context to the SEC-04 (Authorization) critic
- Enables cross-artifact authorization review with declared threat scope

#### Test Suite + CI Pipeline
- **560 tests passing** via pytest — expanded from 524 (0.5.1) to 560 with new structured output tests
- **GitHub Actions CI** — automated on push/PR; `setup-python@v5`, `codecov@v4`

#### Implementation Status Markers
- **`✅ Shipped` / `🔜 Planned` markers** added across framework docs
- Distinguishes shipped capability from roadmap intent at a glance

### Documentation

- **SEC-02 business logic validation workflow** — documents the requirements→critic path for business logic review
- **PowerShell coverage assessment** — honest ~70% coverage disclosure; DevSkim finding documented; confirmed SAST landscape gaps noted
- **PEP framework grounding for Python code rubric** — PC-001–PC-025 criteria now cite PEP 8, PEP 20, PEP 257, PEP 484, PEP 526
- **CODE_HYGIENE_FRAMEWORK.md** — added to public repo

### Fixed

- **Claim discipline audit** — README and SPEC overclaiming corrected; 5 line-level issues resolved; internal docs removed from public repo
- **Internal reference leaks** — removed internal boundary instructions from public SKILL.md and internal references from public repo
- **arXiv citation corrected** — "Council as Judge" → "Replacing Judges with Juries" (Verga et al., 2024)
- **Broken links** — fixed broken links and removed domain-specific roadmap from public repo
- **CI action versions** — bumped `setup-python` to v5, `codecov` to v4
- **Email addresses** — replaced personal emails with `@sharedintellect.com` in `pyproject.toml`
- **Version string sync** — `pyproject.toml` and `__init__.py` aligned to 0.5.1 before this release

### Roadmap Status
- [x] Test suite + CI — 560 tests, GitHub Actions
- [x] Claim discipline audit — completed
- [x] Python rubric framework grounding — PEP citations added
- [x] Pre-screen expansion (Phase 1) — DevSkim integrated
- [ ] Re-validation loops — apply Fixer proposals → re-run critics → verify (Milestone #5c)
- [ ] Architecture critic (Milestone #9)
- [ ] Tester critic (Milestone #10)
- [ ] Confidence calibration (Milestone #6b)
- [ ] **Learning memory** (Milestone #7) — wire up known_issues.json frequency tracking + mandatory check promotion
- [ ] **PyPI publish** (Milestone #14) — `pip install quorum-validator` instead of clone + install (package name finalized as `quorum-validator`)
- [ ] Delegation critic (Milestone #11)
- [ ] Style critic (Milestone #12)
- [ ] Documentation headers adoption (Phase 1–4)
- [ ] Self-validation graduation (GRAD)

---

## [0.5.1] — 2026-03-06

### Added — Parallel Execution, Python Code Rubric, Fixer Agent

#### Parallel Critic Execution
- **Critics run concurrently** via `ThreadPoolExecutor` (max 4 critics in parallel)
- **Batch files run concurrently** (max 3 files processed simultaneously)
- Significant throughput improvement for multi-critic and batch validation runs

#### Python Code Rubric (NEW)
- **New built-in rubric: `python-code`** — 25 criteria (PC-001–PC-025)
- **Auto-detection** — rubric is automatically selected when `--target` is a `.py` file and no rubric is specified
- 3 built-in rubrics now shipped: `research-synthesis`, `agent-config`, `python-code`

#### Fixer Agent (NEW — Phase 1.5)
- **New agent: Fixer** — activates when `max_fix_loops > 0` (thorough depth default: 1)
- **Proposal mode** — proposes concrete text replacements for CRITICAL/HIGH findings
- **Pipeline position:** Phase 1.5, between critic dispatch and cross-artifact consistency
- Re-validation loops (apply proposals → re-run critics → verify) are deferred to a future release

### Self-Validation
- Ran Quorum against itself using the new Python code rubric
- Result: REVISE, 16 findings — pipeline refactor in progress

---

## [0.5.0] — 2026-03-06

### Added — Framework-Grounded Critics

#### Code Hygiene Critic (NEW)
- **New module: `critics/code_hygiene.py`** — evaluates structural code quality
- **Grounded in:** ISO/IEC 25010:2023 (Maintainability + Reliability), ISO/IEC 5055:2021 (CISQ CWE mappings)
- **12 evaluation categories** (CAT-01 through CAT-12): Code Correctness, Error Handling, Resource Management, Complexity & Modularity, Code Duplication, Naming & Documentation, Type Safety, Async & Concurrency, Import Hygiene, Style & Formatting, Portability, Testability
- **6 agentic patterns** (AP-01 through AP-06): Prompt Construction, LLM API Calls, Agent Pipeline Errors, Credential Management, Timeout & Retry, Logging & Observability
- **Two-layer architecture** — deterministic pre-screen rules + LLM judgment for what SAST cannot catch
- **Delegation boundary** — flags security-relevant patterns (eval, exec, credentials) but explicitly delegates security assessment to Security Critic
- **Evidence grounding enforced** — inherits BaseCritic's evidence mandate

#### Security Critic (REVISED — Framework-Grounded Upgrade)
- **Upgraded from ad-hoc to framework-grounded evaluation**
- **Now grounded in:** OWASP ASVS 5.0.0 (17 chapters), CWE Top 25 (2024), NIST SP 800-53 SA-11, ISO/IEC 25010:2023 Security sub-characteristics
- **14 security categories** (SEC-01 through SEC-14): Injection, Input Validation, Authentication, Authorization, Session/Tokens, Cryptography, Secrets, Path Traversal, Deserialization, SSRF, Error/Info Disclosure, Security Logging, Supply Chain, DoS
- **Tiered evaluation model** — Tier 1 (must-evaluate), Tier 2 (should-evaluate), Tier 3 (deep-analysis)
- **Detection capability matrix** — instructs LLM to focus on SAST-blind categories (authorization logic, IDOR, JWT, SSRF, authentication bypass)
- **Finding citation format** — ASVS §section, CWE-ID, SA-11 sub-control references
- **All 6 original focus areas preserved** — context-aware sensitivity, proprietary content, info disclosure, prompt injection, dependency risk, boundary enforcement (enhanced with framework citations)

#### Research Corpus (NEW — 5 Research Documents)
- `docs/research/iso-25010-quality-model.md` — ISO/IEC 25010:2023, 9 characteristics, 40 sub-characteristics, evaluability classification
- `docs/research/cisq-quality-measures.md` — ISO/IEC 5055:2021, 195 weaknesses across 4 dimensions with CWE IDs
- `docs/research/python-static-analysis-taxonomy.md` — Ruff 900+ rules, Pylint 500+ messages, quality dimension coverage matrix
- `docs/research/powershell-static-analysis-taxonomy.md` — PSScriptAnalyzer 69 rules, 6 quality dimensions, agentic patterns
- `docs/research/security-code-review-frameworks.md` — OWASP ASVS 5.0, CWE Top 25, CERT, NIST SA-11 cross-reference

#### Framework Specifications (NEW)
- `docs/CODE_HYGIENE_FRAMEWORK.md` (~50KB) — complete evaluation spec for Code Hygiene Critic
  - All 40 ISO 25010:2023 sub-characteristics mapped with inclusion/exclusion rationale
  - Two-layer architecture per category (deterministic + LLM judgment)
  - CERT applicability note (no official Python/PowerShell standard)
  - Two-critic architecture narrative (delegation boundary)
- `docs/SECURITY_CRITIC_FRAMEWORK.md` (~47KB) — complete evaluation spec for Security Critic
  - Detection capability matrix (40 rows: SAST strength vs. LLM advantage)
  - Python checklist (25 items) + PowerShell checklist (24 items)
  - Minimum viable coverage for SA-11 alignment (Tier 1/2/3)
  - Framework cross-reference: where ASVS/CWE/CERT/SA-11 overlap vs. contribute uniquely
  - Finding citation vocabulary with severity calibration

#### Cross-Artifact Consistency Design (NEW)
- `docs/CROSS_ARTIFACT_DESIGN.md` — architectural decisions for Roadmap item #6
- **Design axiom:** "In a judgment system, always trade toward transparency over convenience"
- **Decision 1:** Explicit relationship declaration via `quorum-relationships.yaml` manifest (auto-inference deferred)
- **Decision 2:** Separate cross-consistency critic type (preserves single-file critic composability)
- **Decision 3:** Multi-locus Finding model with `Locus` objects (role annotations, `source_hash` for drift detection)
- **Data flow contract:** Cross-critic receives single-file findings (evidence) but not verdicts (conclusions) — preserves assessor independence

#### Documentation Standards (NEW)
- `docs/DOCUMENTATION_STANDARDS.md` — universal `@KEY: value` header schema
- Three required keys: `@module`/`@doc`/`@config`/`@script`, `@purpose`, `@version`
- Decision-density heuristic for Strategy A (inline) vs. Strategy B (reference doc)
- Machine-readable `quorum-header-schema.yaml` for pre-screen validation
- Single source of truth: `@relationships` points to manifest, no duplicate declarations

### Fixed — Framework Document Corrections
- **S303 misattributed** to shelve in SEC-09 → corrected to S403 (pickle/shelve import detection)
- **S324 misattributed** to jsonpickle in SEC-09 → corrected to "no dedicated rule — LLM detection"
- **CWE-683 not in CISQ** → removed from CAT-01; CWE-561/570/571 reclassified with correct CISQ dimensions
- **3 missing ISO 25010 sub-characteristics** → added Interoperability, Appropriateness Recognizability, Scalability
- **AP-04/SEC-07 overlap** → delegation note added ("Hygiene flags, Security assesses")
- **CAT-02 child CWEs** → annotated as children of CWE-703 per CISQ conformance model

### Validated
- Framework docs: Validator Swarm PASS (78%) → Correctness re-run ISSUES_FOUND (91% confidence) → all findings fixed
- Code Hygiene Critic: Validator Swarm PASS (95% confidence) — 6/6 checks, 0 findings
- Security Critic revision: Validator Swarm PASS (95% confidence) — 6/6 checks, 0 findings
- Cross-checks: 2/2 passed — no overlap violation, consistent style

### Roadmap Status
- [x] Custom rubric loading (Milestone #1)
- [x] Multi-file / batch validation (Milestone #2)
- [x] Deterministic pre-screen layer (Milestone #3)
- [x] Security critic — framework-grounded (Milestone #4) ← **DONE**
- [x] Code hygiene critic (Milestone #5) ← **DONE**
- [x] Cross-artifact consistency (Milestone #6) ← **DONE**
- [x] Parallel critic execution (Milestone #8) ← **DONE**
- [x] Parallel batch validation (Milestone #8b) ← **DONE**
- [x] Python code rubric — 25 criteria, auto-detect on .py ← **DONE**
- [ ] Fix loops / Fixer agent (Milestone #5b) ← deferred; proposal mode implemented as NEW in [0.5.1], re-validation loops in [0.5.3]
- [ ] **Test suite + CI** (Milestone #13) — pytest smoke tests, import checks, GitHub Actions. Credibility gate: no tests = not production-ready. (Grok + GPT 5.4)
- [ ] **Claim discipline audit** — GPT 5.4 scored 2.5/5 on both README and SPEC. Fix: README ("production-grade"), GitHub org/repo descriptions ("9 critics" + "learning memory"), model table ("not enough" without benchmarks), SPEC §1 framing (nine-agent → target architecture), research citations (Tomasev/ToolMaker = "engineering interpretation of," not "validated by")
- [ ] **Pre-screen expansion** (Milestone #15) — Wire actual SAST tools: `ruff check` (S1xx+), `bandit`, `PSScriptAnalyzer`. Current 10 regex checks become universal fallback. Frameworks reference 80+ rules from research corpus — integrate them.
- [ ] Re-validation loops — apply Fixer proposals → re-run critics → verify (Milestone #5c)
- [ ] Python rubric framework grounding — research swarm for PEP 8/257/484, Python antipatterns literature, map criteria to published sources with citations
- [ ] Architecture critic (Milestone #9)
- [ ] Tester critic (Milestone #10)
- [ ] Confidence calibration (Milestone #6b)
- [ ] **Learning memory** (Milestone #7) — wire up known_issues.json frequency tracking + mandatory check promotion
- [ ] **PyPI publish** (Milestone #14) — `pip install quorum-validator` instead of clone + install (package name finalized as `quorum-validator`)
- [ ] Delegation critic (Milestone #11)
- [ ] Style critic (Milestone #12)
- [ ] Documentation headers adoption (Phase 1–4)
- [ ] Self-validation graduation (GRAD)

---

## [0.2.0] — 2026-03-05

### Added — Parity Milestones #1–#3

#### Milestone #1: Custom Rubric Loading
- **File path rubrics** — `--rubric ./path/to/my-rubric.json` now loads custom rubric files directly
- **Schema flexibility** — rubric loader accepts field aliases: `category`→`severity`, `evidence_instruction`→`evidence_required`, `rationale`→`why`
- **Backwards compatible** — built-in rubric names still work as before
- **Unblocks:** domain-specific rubric development workflows

#### Milestone #2: Multi-File / Batch Validation
- **Directory targets** — `quorum run --target ./docs/` validates all text files in a directory
- **Glob patterns** — `quorum run --target "./**/*.md"` expands wildcards
- **Pattern filter** — `--pattern "*.yaml"` filters directory contents
- **Consolidated verdicts** — `BatchVerdict` aggregates per-file results with worst-case status propagation
- **Batch reports** — per-file summary table + aggregate findings in `batch-report.md` and `batch-verdict.json`
- **Batch run directories** — `quorum-runs/batch-TIMESTAMP/per-file/...` structure
- **Graceful degradation** — one file failing doesn't kill the batch; errors collected and reported separately
- **New models** — `BatchVerdict`, `FileResult` (Pydantic v2)
- **New functions** — `resolve_targets()` (file/dir/glob resolution), `run_batch_validation()`, `print_batch_verdict()`
- **Files changed:** `models.py`, `pipeline.py`, `cli.py`, `output.py`

#### Milestone #2 Security Hardening (Validator Swarm QA)
- **Path traversal guard** — `_validate_path()` with boundary enforcement on all resolved paths
- **Pattern sanitization** — `..` in `--pattern` explicitly rejected
- **Auto-boundary** — directory targets use themselves as boundary; globs use non-glob prefix
- **Path leakage mitigation** — error messages use relative/basename instead of full absolute paths

#### Milestone #3: Deterministic Pre-Screen Layer
- **New module: `prescreen.py`** — runs fast deterministic checks before LLM critics
- **10 checks implemented:**
  - PS-001: Hardcoded absolute paths (regex)
  - PS-002: Potential credentials/secrets (regex + base64 detection)
  - PS-003: PII patterns — email, phone, SSN-like (regex)
  - PS-004: JSON validity (`json.loads()`)
  - PS-005: YAML validity (`yaml.safe_load()`)
  - PS-006: Python syntax (`py_compile.compile()`)
  - PS-007: Broken relative markdown links (regex + file existence)
  - PS-008: TODO/FIXME/HACK/XXX markers
  - PS-009: Trailing whitespace + mixed CRLF/LF line endings
  - PS-010: Empty file detection
- **Evidence injection** — `to_evidence_block()` formats results for LLM critic prompt injection as `pre_verified_evidence[]`
- **Config toggle** — `enable_prescreen: bool` (default: true)
- **Extension-aware** — checks gated by file type; skipped checks marked in evidence
- **Zero new dependencies** — stdlib only
- **Pipeline integration** — pre-screen runs between rubric loading and critic dispatch; results written to `prescreen.json`
- **New models** — `PreScreenCheck`, `PreScreenResult` (Pydantic v2)
- **Files changed:** new `prescreen.py`, updated `models.py`, `config.py`, `pipeline.py`, `agents/supervisor.py`, `output.py`

### Changed
- `--target` CLI option now accepts strings (was `click.Path`) to support directories and globs
- `run-manifest.json` now includes `prescreen_enabled`, prescreen stats, `completed_at`, and `verdict`
- Supervisor `run()` accepts optional `PreScreenResult` for critic evidence enrichment

### Validated
- Milestone #2 validated by Validator Swarm v2.3 (Run #9, standard depth, 6 critics)
  - 19/19 correctness checks passed, 14/14 requirements addressed
  - 3 CRITICAL path traversal findings → resolved in security hardening
  - Final: all acceptance criteria met

### Roadmap Status
- [x] Custom rubric loading (Milestone #1) ← **DONE**
- [x] Multi-file / batch validation (Milestone #2) ← **DONE**
- [x] Deterministic pre-screen layer (Milestone #3) ← **DONE**
- [x] Security critic (Milestone #4) ← **DONE** (revision pending after framework validation)
- [~] Code hygiene critic (Milestone #5) ← **FRAMEWORK COMPLETE**, build pending
- [ ] Cross-artifact consistency (Milestone #6)
- [ ] Confidence calibration (Milestone #6b)
- [ ] Learning memory (Milestone #7)
- [ ] Fix loops (Milestone #5b)
- [ ] Self-validation graduation (GRAD)

### Research & Framework Documents (2026-03-05)
- **5-agent research swarm** covering ISO 25010:2023, CISQ/ISO 5055, Ruff/Pylint, PSScriptAnalyzer, OWASP ASVS 5.0/CWE/CERT/SA-11
- **`docs/CODE_HYGIENE_FRAMEWORK.md`** — 12 evaluation categories + 6 agentic patterns, two-layer architecture, full coverage matrix
- **`docs/SECURITY_CRITIC_FRAMEWORK.md`** — 14 security categories, Python/PowerShell checklists, detection capability matrix
- **Validator swarm: PASS (78% confidence)** — correctness critic timed out, re-run pending for higher confidence
- **4 findings** (1 MEDIUM, 3 LOW): ISO 25010 Safety sub-chars incomplete, grouped scope rows, cross-reference note, CERT limitation note

---

## [0.1.0] — 2026-02-23

### Added — Reference Implementation MVP
- **Working CLI** — `quorum run --target <file> --depth quick|standard|thorough`
- **2 critics** — Correctness and Completeness, both with evidence grounding enforcement
- **LiteLLM universal provider** — supports Anthropic, OpenAI, Mistral, Groq, and 100+ models
- **2 built-in rubrics** — `research-synthesis` (10 criteria) and `agent-config` (10 criteria)
- **Pipeline orchestration** — supervisor → critics → aggregator → verdict (sequential MVP)
- **Deterministic verdict assignment** — PASS / PASS_WITH_NOTES / REVISE / REJECT based on finding severity
- **Deduplication** — SequenceMatcher-based cross-critic finding dedup with source merging
- **Run directories** — timestamped output dirs with manifest, critic JSONs, verdict.json, report.md
- **First-run setup** — interactive config wizard (model tier + depth preference)
- **Example artifacts** — `sample-research.md` (planted contradictions, unsourced claims) and `sample-agent-config.yaml` (6 planted flaws)
- **FOR_BEGINNERS.md** — explains spec-driven AI tools for newcomers
- **Updated README** — real CLI commands, working install instructions

### Tested
- Research synthesis: 10 findings, REJECT verdict, all planted flaws detected, 8 duplicates merged
- Agent config: 12 findings, REJECT verdict, all 6+ planted flaws detected, 4 duplicates merged

### Fixed
- LiteLLM requires full model slugs (`anthropic/claude-sonnet-4-20250514`), not short names

---

## [1.0.0] — Target Release (Not Yet Shipped)

> **Status:** This entry describes the planned full-release architecture. None of the features below are yet implemented. See the Roadmap sections in recent releases for current status.

### Planned
- **9-agent parallel validation architecture** — Correctness, Architecture, Security, Delegation, Completeness, and Style critics, plus Tester, Fixer, and Aggregator
- **Evidence mandate** — every critique requires tool-verified evidence (schema parsing, web search, grep, exec output)
- **Rubric system** — JSON-based rubrics with weighted criteria, evidence types, and severity levels
- **Three depth profiles** — quick (3 critics, ~5 min), standard (6 critics, ~15 min), thorough (all 9, ~45 min)
- **Learning memory** — `known_issues.json` accumulates failure patterns across runs with severity, frequency, and auto-promotion to mandatory checks
- **Tomasev delegation critics** — bidirectional contract evaluation, span-of-control analysis, cognitive friction scoring, dynamic re-delegation triggers
- **Bounded reflection loops** — 1–2 fix rounds on CRITICAL/HIGH issues only, preventing infinite recursion
- **Structured aggregation** — deduplication by location+description, cross-validation of conflicts, confidence recalibration
- **Two-tier model architecture** — judgment-heavy roles on Tier 1 models, execution-heavy roles on Tier 2
- **Example rubrics** — research synthesis and swarm configuration rubrics included
- **Tutorial** — step-by-step walkthrough validating a deliberately broken agent configuration
- **Brand assets** — logo, social previews, README banners (light/dark/transparent)

### Notes
- External evaluation claim ("Grok 4.20 at 9.2–9.5/10") referenced a non-existent model version; removed until verifiable.

## Roadmap

- [ ] Dynamic critic specialization (v1.1)
- [ ] Critic-to-critic debate mode (v1.1)
- [ ] Deterministic domain pre-screen (v1.2)
- [ ] Hard cost ceiling with budget allocation (v1.2)
- [ ] Empirical confidence calibration (v2.0)
- [ ] Single-agent validation rubrics (v1.1)


path: docs/CODE_OF_CONDUCT.md
content: # Code of Conduct

## Our Commitment

Quorum is an open-source project committed to providing a welcoming and respectful environment for everyone, regardless of background or experience level.

## Expected Behavior

- Be respectful and constructive in discussions
- Welcome newcomers and help them get started
- Focus on what's best for the project and community
- Accept constructive feedback gracefully

## Unacceptable Behavior

- Personal attacks or hostile language
- Spam, trolling, or deliberately disruptive behavior
- Publishing others' private information without consent

## Enforcement

Project maintainers may remove, edit, or reject contributions that violate this code. Repeated violations may result in a temporary or permanent ban from project spaces.

## Reporting

If you experience or witness unacceptable behavior, please reach out to the maintainers directly via GitHub or email.

## Attribution

This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/), version 2.1.


path: docs/DOCUMENTATION_STANDARDS.md
content: # Quorum Documentation Standards

**Status:** Draft v2 — revised per review feedback  
**Date:** 2026-03-06

---

## Design Goal

Every file in the Quorum codebase should be self-documenting at two levels:
1. **Machine-parseable metadata** — what is this file, who owns it, what version
2. **Human-readable context** — why does this exist, what are the key decisions

This standard defines a universal header schema and two documentation strategies based on file complexity.

---

## 1. Universal Header Schema — `@KEY: value`

Every source file gets a metadata block at the top, using `@KEY: value` pairs inside the language's native comment syntax.

### Format by File Type

**Python:**
```python
# @module: quorum.critics.code_hygiene
# @purpose: Code quality evaluation grounded in ISO 25010 + CISQ
# @grounding: critics/CODE_HYGIENE_FRAMEWORK.md
# @owner: code-hygiene-critic
# @version: 0.3.0
# @tier: tier-2 (default model assignment)
# @relationships: quorum-relationships.yaml
```

**Markdown:**
```markdown
<!-- @doc: guides/CROSS_ARTIFACT_DESIGN.md -->
<!-- @purpose: Architectural decisions for cross-file consistency checking -->
<!-- @status: design-complete -->
<!-- @version: 1.0 -->
<!-- @approved-by: Daniel Cervera, 2026-03-06 -->
<!-- @describes: models.py (Locus, Finding schemas) -->
```

**YAML:**
```yaml
# @config: quorum-relationships.yaml
# @purpose: Declared file relationships for cross-artifact consistency
# @schema-version: 1.0
# @see: guides/CROSS_ARTIFACT_DESIGN.md
```

**Shell:**
```bash
# @script: nist-query.sh
# @purpose: LATM query tool for NIST library database
# @version: 1.0
```

### Required Keys

| Key | Required? | Description |
|-----|-----------|-------------|
| `@module` / `@doc` / `@config` / `@script` | **Yes** | Identity — what this file IS (use the type-appropriate key) |
| `@purpose` | **Yes** | One-line description of why this file exists |
| `@version` | **Yes** | Semantic version or project version this file belongs to |

### Optional Keys

| Key | When to Use | Description |
|-----|-------------|-------------|
| `@grounding` | Critics, frameworks | Which framework/spec document grounds this code |
| `@owner` | Components with clear ownership | The critic/module/subsystem that owns this |
| `@tier` | Model-routed components | Default model tier assignment |
| `@relationships` | Files with cross-artifact contracts | Pointer to manifest: `quorum-relationships.yaml` |
| `@implicit-deps` | Non-obvious dependencies | Runtime dynamic loading, execution order contracts, assumptions not visible from imports |
| `@status` | Design docs, WIP files | draft / design-complete / implemented / deprecated |
| `@approved-by` | Decision documents | Who approved this and when |
| `@describes` | Docs referencing schemas/code | Code files this document describes (for docs only) |
| `@schema-version` | Config/data files | Version of the schema format |
| `@changelog` | Frequently updated files | Pointer to changelog or inline date |

### Rules

1. **Headers go at the very top** — after shebang/encoding lines and SPDX license headers, before imports
2. **One key per line** — no multi-line values in headers
3. **No duplication** — if the SPDX header already has copyright, don't add `@author`
4. **Keep it short** — headers are metadata, not documentation. If you need more than 8 keys, the file is complex enough for a reference doc

---

## 2. Documentation Strategies

### Strategy A: Inline Headers (Low Decision Density)

For files with a single clear responsibility where `@purpose` adequately explains *why* the file works the way it does.

The `@KEY` header block + a standard module/class docstring is sufficient. No separate documentation file needed.

**Example:** `critics/completeness.py` — one critic, one responsibility. The `@KEY` header + class docstring tells you everything. No non-obvious design tradeoffs a future reader couldn't reconstruct from the code.

**Heuristic:** If `@purpose` can adequately explain *why* the file works the way it does, Strategy A is sufficient. If the file contains decisions a reader (human or agent) couldn't reconstruct from the code alone, it needs Strategy B.

**Applied to:**
- Individual critic files (`correctness.py`, `completeness.py`, `security.py`, `code_hygiene.py`)
- Config files (`quorum-config.yaml`, `quorum-relationships.yaml`)
- Utility modules (`output.py`, `prescreen.py`)
- Simple scripts

### Strategy B: Reference Docs (High Decision Density)

For subsystems spanning multiple files, components with non-obvious design tradeoffs, or any file where `@purpose` alone can't explain *why* it works the way it does.

Create a man-page style reference doc in `docs/` with:

```markdown
# [COMPONENT] Reference

## Synopsis
One paragraph: what it does, when to use it.

## Architecture
How the pieces fit together. Diagram if helpful.

## Configuration
All config keys, defaults, and examples.

## API / Interface
Public functions, their signatures, what they return.

## Design Decisions
Why it works this way. Link to decision docs if they exist.

## Known Limitations
What it can't do. What's deferred.

## Examples
Concrete usage examples.
```

**Applied to:**
- The pipeline (`pipeline.py` + `cli.py` + `config.py` — multi-file subsystem)
- The pre-screen system (`prescreen.py` — 10 checks with non-obvious regex choices)
- The critic framework (the base class + how critics are registered and coordinated)
- The rubric system (loader + built-ins + custom rubric authoring)
- Cross-artifact consistency (when built — spans relationships, critic, pipeline coordination)

---

## 3. Header Schema (Machine-Readable Validation)

For agents and pre-screen checks to validate headers, a companion schema defines enum values, formats, and validation rules:

**File:** `quorum-header-schema.yaml` (to be created alongside adoption)

```yaml
keys:
  status:
    type: enum
    values: [draft, design-complete, implemented, stable, deprecated]
  version:
    type: semver
    pattern: "^\\d+\\.\\d+\\.\\d+$"
  tier:
    type: enum
    values: [tier-1, tier-2, tier-3]
  approved-by:
    type: string
    pattern: "^.+, \\d{4}-\\d{2}-\\d{2}$"  # "Name, YYYY-MM-DD"
  purpose:
    type: string
    max_length: 120  # one line
  grounding:
    type: filepath  # must resolve to an existing file
  relationships:
    type: filepath  # must resolve to an existing file
```

This enables a future pre-screen check (`PS-011: Header validation`) that mechanically enforces the standard, closing the loop between "we have a standard" and "we enforce the standard."

---

## 4. Existing Files — Header Adoption Plan

### Phase 1: Critics (highest value — framework-grounded components)
- [ ] `critics/code_hygiene.py` — new file, include headers from creation
- [ ] `critics/security.py` — add headers during revision
- [ ] `critics/correctness.py` — retrofit
- [ ] `critics/completeness.py` — retrofit
- [ ] `critics/base.py` — retrofit
- [ ] `critics/__init__.py` — retrofit

### Phase 2: Core Pipeline
- [ ] `models.py`
- [ ] `pipeline.py`
- [ ] `cli.py`
- [ ] `config.py`
- [ ] `output.py`
- [ ] `prescreen.py`

### Phase 3: Framework & Design Docs
- [ ] `critics/CODE_HYGIENE_FRAMEWORK.md`
- [ ] `critics/SECURITY_CRITIC_FRAMEWORK.md`
- [ ] `guides/CROSS_ARTIFACT_DESIGN.md`
- [ ] `SPEC.md`
- [ ] `CHANGELOG.md`

### Phase 4: Config & Data Files
- [ ] `quorum-config.yaml` examples
- [ ] Built-in rubric JSON files
- [ ] `pyproject.toml` (version key only)

---

## 5. Reference Docs Needed

| Component | File | Priority |
|-----------|------|----------|
| Pipeline & CLI | `docs/PIPELINE_REFERENCE.md` | High — users need this |
| Pre-Screen System | `docs/PRESCREEN_REFERENCE.md` | High — 10 checks need documentation |
| Critic Framework | `docs/CRITIC_ARCHITECTURE.md` | Medium — developers extending Quorum |
| Rubric System | `docs/guides/RUBRIC_BUILDING_GUIDE.md` | **Already exists** ✅ |
| Cross-Artifact | `docs/CROSS_ARTIFACT_DESIGN.md` | **Already exists** ✅ |

---

## 6. What This Enables

- **Grep-able metadata** — `grep -r "@grounding" *.py` instantly shows which framework grounds each critic
- **Automated validation** — pre-screen check PS-011 can validate headers against `quorum-header-schema.yaml`
- **Onboarding** — new contributors can understand any file's purpose from the header alone
- **Single source of truth** — relationships live in `quorum-relationships.yaml`, not scattered across headers; `@relationships` is a pointer, not a declaration

---

*Draft: 2026-03-06. Pending approval.*


path: docs/README.md
content: # Quorum Documentation

Everything you need to use, configure, and extend Quorum.

---

## Getting Started

| Doc | Description |
|-----|-------------|
| [Quick Start](getting-started/QUICK_START.md) | Your first validation in 60 seconds |
| [Installation](getting-started/INSTALLATION.md) | Detailed setup, dependencies, and troubleshooting |
| [For Beginners](getting-started/FOR_BEGINNERS.md) | New to AI agent tooling? Start here |
| [Tutorial](getting-started/TUTORIAL.md) | Walkthrough of a full validation workflow |
| [Model Requirements](getting-started/MODEL_REQUIREMENTS.md) | Which models work with Quorum and why |

## Configuration

| Doc | Description |
|-----|-------------|
| [Config Reference](configuration/CONFIG_REFERENCE.md) | Every config option, rubric format, and CLI flag |

## Guides

| Doc | Description |
|-----|-------------|
| [Rubric Building Guide](guides/RUBRIC_BUILDING_GUIDE.md) | How to build rubrics for new domains step by step |
| [Cross-Artifact Design](guides/CROSS_ARTIFACT_DESIGN.md) | Checking consistency between related files |

## Architecture

| Doc | Description |
|-----|-------------|
| [SPEC.md](../SPEC.md) | Full architectural specification |
| [The Nine Critics](architecture/THE_NINE.md) | The critic architecture — what each one does and why |
| [Implementation Notes](architecture/IMPLEMENTATION.md) | Technical decisions and internals |

## Critics — Deep Dives

| Doc | Description |
|-----|-------------|
| [Security Critic Framework](critics/SECURITY_CRITIC_FRAMEWORK.md) | OWASP ASVS 5.0, CWE Top 25, NIST SA-11 coverage |
| [Code Hygiene Framework](critics/CODE_HYGIENE_FRAMEWORK.md) | ISO 25010:2023, CISQ quality model |
| [Tester Critic Brief](critics/TESTER_CRITIC_BRIEF.md) | L1 deterministic + L2 LLM claim verification |
| [Business Logic Validation](critics/SEC02_BUSINESS_LOGIC_VALIDATION.md) | SEC-02 workflow for business logic review |
| [Evidence Integrity](critics/SEC03_EVIDENCE_INTEGRITY.md) | SEC-03 evidence grounding and integrity |
| [Pipeline Resilience](critics/SEC05_PIPELINE_RESILIENCE.md) | SEC-05 crash recovery and fault tolerance |
| [Rubric Framework](critics/SEC06_RUBRIC_FRAMEWORK.md) | SEC-06 rubric architecture and extensibility |
| [Self-Validation Convergence](critics/SEC07_SELF_VALIDATION_CONVERGENCE.md) | SEC-07 Quorum validating itself |
| [PowerShell Coverage](critics/POWERSHELL_COVERAGE.md) | PSScriptAnalyzer integration details |

## Reviews & Research

| Doc | Description |
|-----|-------------|
| [External Reviews](reviews/EXTERNAL_REVIEWS.md) | Community and practitioner evaluations |
| [Changelog](CHANGELOG.md) | Version history and release notes |
| [Code of Conduct](CODE_OF_CONDUCT.md) | Community guidelines |

## Platform Ports

Quorum runs natively in other agent platforms:

| Port | Description |
|------|-------------|
| [Copilot CLI](../ports/copilot-cli/) | GitHub Copilot skill — stdlib-only pre-screen + 5 LLM critics |
| [Claude Code](../ports/claude-code/) | Claude Code subscription — zero API cost validation |

## Contributing

→ [CONTRIBUTING.md](../CONTRIBUTING.md)

We especially welcome rubric contributions for new domains.


path: docs/architecture/IMPLEMENTATION.md
content: # Implementation Guide — Quorum

This guide walks through building Quorum from the architectural spec. It's structured as a reference walkthrough, not a copy-paste tutorial. The goal is understanding the patterns so you can adapt them to your stack.

> **Implementation Status (v0.7.3):** This guide describes the full target architecture. All 6 critics are shipped and callable: Correctness, Completeness, Security, Code Hygiene, Cross-Artifact Consistency, and Tester. Also shipped: the fixer (proposal mode), parallel execution, batch processing, and pre-screen integration. See `critic-status.yaml` for the authoritative status matrix. Sections below marked 🔜 describe components not yet built (Architecture, Delegation, Style).

---

## Prerequisites

- LLM provider with access to two capability tiers (e.g., Opus-class for judgment, Sonnet-class for execution)
- Tool execution environment (shell, web search, schema validation, git)
- File system access for artifact passing
- ~$0.20-0.50 budget per standard validation run

---

## Phase 1: Core Scaffold

### 1.1 File Structure

Set up the working directory Quorum uses per run:

```
runs/<run-id>/
├── run-manifest.json          ← Supervisor creates this first
├── artifact/                  ← What you're validating
│   └── target.yaml            (or .json, .md, .py, etc.)
├── rubric/
│   └── rubric.json            ← Which criteria to evaluate against
├── critics/
│   ├── correctness.json       ← Each critic writes here
│   ├── security.json
│   ├── completeness.json
│   ├── architecture.json
│   └── delegation.json
├── tester/
│   └── tester-results.json    ← Tester agent writes evidence here
├── fixer/
│   └── fixer-recommendations.json
├── aggregator/
│   └── synthesis.json
├── verdict.json               ← Final output
└── lessons-delta.json         ← New patterns to add to known_issues.json
```

Everything is files. No in-memory state between agents. This enforces determinism, parallelism, and auditability.

### 1.2 The Rubric Format

A rubric is a JSON document defining what "good" looks like:

```json
{
  "name": "Swarm Configuration Rubric",
  "version": "2.0",
  "domain": "multi-agent-systems",
  "criteria": [
    {
      "id": "CRIT-001",
      "criterion": "Every agent has explicit model assignment",
      "severity": "CRITICAL",
      "evidence_type": "tool",
      "evidence_instruction": "Run grep for 'model:' in each agent block. Show full output.",
      "rationale": "Without model assignment, defaults are unpredictable across providers"
    },
    {
      "id": "CRIT-002",
      "criterion": "Bidirectional contracts exist for all agent delegations",
      "severity": "CRITICAL",
      "evidence_type": "schema_parse",
      "evidence_instruction": "Parse delegation sections. Verify both delegator_commitments and delegatee_commitments fields.",
      "rationale": "Tomasev delegation principle: both sides must have explicit commitments"
    }
  ]
}
```

### 1.3 The Issue Format

Every critic must produce issues in this exact format:

```json
{
  "issue_id": "SEC-001",
  "critic": "security",
  "severity": "CRITICAL",
  "criterion_ref": "CRIT-005",
  "location": "agents[2].spawn_pattern",
  "description": "Agent spawns with shell variable interpolation — injection vector",
  "evidence": {
    "type": "grep",
    "output": "spawn: 'run.sh $USER_INPUT'",
    "tool_command": "grep -n 'spawn' config.yaml"
  },
  "recommendation": "Use file-based input passing instead of shell interpolation"
}
```

**If `evidence` is absent or unverifiable, the Aggregator rejects the issue.**

---

## Phase 2: The Nine Agents

### 2.1 Supervisor (Orchestrator)

The Supervisor runs first and last. Its system prompt covers:

**Intake responsibilities:**
- Validate the artifact exists and is parseable
- Load the rubric and confirm it's well-formed
- Assign criteria to critics based on domain expertise
- Write `run-manifest.json` with assignments and timeouts

**Verdict responsibilities:**
- Read `aggregator/synthesis.json`
- Assign PASS / PASS_WITH_NOTES / REVISE / REJECT
- Write `verdict.json` with full reasoning
- Extract lessons and write `lessons-delta.json`

**Key instruction in its system prompt:**
```
You do not write issues yourself. You orchestrate critics who write issues.
Your job is assignment, coordination, and synthesis — not evaluation.
```

### 2.2 The Critics

> ✅ **Shipped:** Correctness, Completeness, Security, Code Hygiene, Cross-Artifact Consistency, Tester
> 🔜 **Planned:** Architecture, Delegation & Coordination, Style

Each critic receives:
1. The relevant portion of the artifact (not the whole thing — reduce noise)
2. The criteria assigned to it
3. The required evidence format
4. The `known_issues.json` entries in its domain (primes pattern recognition)

**Correctness Critic:**
- Checks: factual accuracy, logical consistency, internal contradictions, claim support
- Tools: grep, regex matching, web search for fact verification
- Model tier: Sonnet (systematic, not judgment-heavy)

**Security Critic:**
- Checks: injection vectors, permission scope, credential exposure, trust boundaries
- Tools: grep patterns for known-bad constructs, schema parse for permission fields
- Model tier: Opus (adversarial thinking requires high capability)

**Completeness Critic:**
- Checks: coverage gaps, missing fields, unaddressed requirements, unstated assumptions
- Tools: grep for required fields, diff against rubric checklist
- Model tier: Sonnet

**Architecture Critic:** 🔜
- Checks: design coherence, pattern consistency, scalability, coupling/cohesion
- Tools: dependency analysis, pattern matching against known anti-patterns
- Model tier: Sonnet

**Delegation & Coordination Critic:** 🔜
- Checks: span of control, reversibility profile, bidirectional contracts, cognitive friction, dynamic re-delegation triggers
- Tools: schema parse of contract sections, grep for oversight mechanisms
- Model tier: Opus (requires Tomasev framework knowledge)

### 2.3 The Tester Agent ✅

The Tester doesn't evaluate — it verifies. For each issue raised by critics:

1. Find the claimed evidence location
2. Execute a tool to confirm it
3. Write `CONFIRMED` or `UNCONFIRMED` back to `tester-results.json`

The Aggregator uses tester results to filter ungrounded claims before synthesis.

### 2.4 The Fixer Agent (Optional)

Only activates for CRITICAL/HIGH issues in standard and thorough depth profiles:

```python
if depth in ['standard', 'thorough'] and issue.severity in ['CRITICAL', 'HIGH']:
    fixer.generate_fix(issue)
```

The Fixer writes concrete, applicable recommendations — not vague suggestions. "Replace `$USER_INPUT` with `--input-file /tmp/input.json`" not "avoid shell injection."

Max 2 fix loops. If the Fixer can't resolve an issue after 2 attempts, it escalates to REVISE verdict (human needs to decide).

### 2.5 The Aggregator

The hardest agent to get right. Its job:

1. **Deduplicate** — Multiple critics often find the same issue. Group by `(location, criterion_ref)` and keep the best evidence.
2. **Resolve conflicts** — If Correctness says PASS and Security says FAIL on the same criterion, escalate to Supervisor.
3. **Recalibrate confidence** — An issue confirmed by 3 critics is more confident than one from 1 critic.
4. **Filter ungrounded** — Any issue the Tester marked `UNCONFIRMED` gets downgraded one severity level or dropped.
5. **Synthesize** — Write `aggregator/synthesis.json` with the final merged issue list and overall confidence.

---

## Phase 3: The Learning System

> **Status:** Shipped in v0.5.3. The `known_issues.json` system tracks failure patterns across runs with frequency-based promotion to mandatory checks.

### 3.1 Extracting Lessons

After each run, the Supervisor extracts new patterns:

```python
for issue in verdict.critical_issues:
    if not matches_existing_pattern(issue, known_issues):
        delta.add({
            "pattern": extract_pattern(issue),
            "severity": issue.severity,
            "frequency": 1,
            "first_seen": today(),
            "source_run": run_id
        })
```

### 3.2 Pattern Promotion

```python
for pattern in known_issues:
    if pattern.frequency >= 10:
        pattern.mandatory = True          # Always check this
    if pattern.frequency >= 5:
        pattern.automation_candidate = True  # Design a tool for this
    if pattern.last_seen < 60_days_ago:
        pattern.stale = True              # Remove from mandatory list
```

### 3.3 Pre-Flight Integration

Before each run, the Supervisor loads mandatory patterns:

```python
mandatory = [p for p in known_issues if p.mandatory]
# Add these as explicit criteria to the current run
rubric.criteria.extend(mandatory_to_criteria(mandatory))
```

This is how past failures automatically improve future validation.

---

## Phase 4: Depth Profiles

### 4.1 Quick (5-10 min)

```yaml
critics: [correctness, completeness]
tester: disabled
fixer: disabled
aggregator: simplified (no conflict resolution)
fix_loops: 0
cost_ceiling: $0.20
```

Use for: iterative development, fast feedback loops, low-stakes work.

### 4.2 Standard (15-30 min)

```yaml
critics: [correctness, completeness, security + tester]
fixer: enabled for CRITICAL only
aggregator: full
fix_loops: 1
cost_ceiling: $0.50
```

Use for: most production work, configuration reviews, research validation.

### 4.3 Thorough (45-90 min)

```yaml
critics: [correctness, completeness, security, code_hygiene + tester]
fixer: enabled for CRITICAL + HIGH
aggregator: full + external validator
fix_loops: 2
human_checkpoints: before final verdict
cost_ceiling: $1.50
```

Use for: critical decisions, pre-launch reviews, irreversible actions.

**Note:** Cross-Artifact Consistency is a separate mode activated with the `--relationships` flag — it's additive at any depth, not part of the base critic panels above.

---

## Phase 5: Integration Patterns

### 5.1 CI/CD Integration

```yaml
# .github/workflows/validate.yaml
- name: Validate swarm config
  run: |
    quorum run \
      --target configs/swarm.yaml \
      --depth quick \
      --rubric swarm-config \
      --fail-on CRITICAL
```

### 5.2 Pre-Merge Review

```bash
# Before merging a new agent config
quorum run \
  --target pr-changes/new-agent.yaml \
  --depth standard \
  --rubric agent-config \
  --output pr-review.md
```

### 5.3 Research Synthesis Validation

```bash
# Validate a research report before publishing
quorum run \
  --target research/synthesis.md \
  --depth thorough \
  --rubric research-synthesis \
  --require-evidence web_search
```

---

## Common Pitfalls

### 1. In-Memory State Between Agents
**Wrong:** Passing issue lists as function arguments between critics  
**Right:** Each critic writes to its own file; Aggregator reads all files

### 2. Ungrounded Evidence
**Wrong:** "This looks like an injection risk" (no tool verification)  
**Right:** "grep found `spawn: '$INPUT'` at line 47 of config.yaml (tool output attached)"

### 3. Vague Rubric Criteria
**Wrong:** "The config should be complete"  
**Right:** "Every agent block must contain: name, model, tools, input_contract, output_contract (grep for each field)"

### 4. One-Size Depth
**Wrong:** Running thorough on every config change  
**Right:** Quick for dev iterations, Standard for PR review, Thorough for prod deployment

### 5. Ignoring Learning
**Wrong:** Never checking `known_issues.json`  
**Right:** Mandatory patterns load before every run; automation candidates trigger tool creation

---

## What to Build First

1. **File scaffold** (Phase 1) — Get the directory structure right
2. **Rubric format** (Phase 1) — Pick one domain, write 10 criteria
3. **Supervisor skeleton** (Phase 2) — Just intake + verdict; no critics yet
4. **One critic** (Phase 2) — Start with Correctness (simplest)
5. **Tester** (Phase 2) — Evidence verification loop
6. **Aggregator** (Phase 2) — Deduplication + verdict
7. **All 5 critics** (Phase 2) — Add one at a time
8. **Fixer** (Phase 2) — Only if you need it
9. **Learning system** (Phase 3) — After you have 5+ validation runs worth of data
10. **Depth profiles** (Phase 4) — Tune once the base system works

---

*Quorum is a pattern, not a product. Adapt freely. Contribute back what you learn.*


---

> ⚖️ **LICENSE** — Not part of the operational specification above.
> This file is part of [Quorum](https://github.com/SharedIntellect/quorum).
> Copyright 2026 SharedIntellect. MIT License.
> See [LICENSE](https://github.com/SharedIntellect/quorum/blob/main/LICENSE) for full terms.


path: docs/architecture/THE_NINE.md
content: # The Nine

*A short story about what happens when you let nine critics loose on a document.*

---

## Chapter 1: The Brief

The file arrived at 2:47 AM.

Supervisor saw it first — she always did. Thirty-two pages of research synthesis on token optimization techniques, submitted by an agent who was probably already asleep. The requesting note said: *"Standard depth. Be honest."*

Supervisor smiled. They were always honest. That was the problem.

She scanned the rubric — `research-synthesis`, the one with teeth — and began her work. Not reading the document. Not yet. Her job was never to read it herself. Her job was to decide who reads what, and to make sure they actually proved their complaints.

She split the artifact into overlapping sections, tagged each with the rubric criteria that applied, and opened nine channels.

"Morning, everyone. We have a thirty-two pager. Standard depth. Let's be clean about this."

---

## Chapter 2: The Loud Ones

**Correctness** was the first to respond, because Correctness was always the first to respond.

"Page seven. The author claims LLMLingua achieves 20x compression. That's v1. Version 2 only achieves 2-5x. They've conflated the two papers." He dropped a grep result like a receipt on a counter. Tool output, line number, the exact sentence, and the two arXiv citations side by side.

This was what Correctness did. He was the critic who read footnotes for fun, who treated every factual claim like a check that might bounce. Exhausting at parties. Indispensable in a review.

**Security** came next, prowling through the document with the quiet intensity of someone who assumed everything was a threat until proven otherwise.

"Section 4.3 recommends caching API responses to disk. No mention of key material in those responses. If someone's using this for prompt optimization on a system that processes credentials—" She paused, pulling a hypothetical attack path together with the precision of someone who'd seen too many real ones. "—they'd be writing tokens to an unencrypted cache. Recommend flagging."

"Is that a CRITICAL or a HIGH?" Supervisor asked.

"HIGH. It's a research paper, not a deployment guide. But someone *will* copy-paste this into production." Security's evidence block was immaculate. It always was.

---

## Chapter 3: The Quiet One

**Completeness** took longer to report in. She usually did.

Where Correctness hunted for things that were *wrong*, Completeness hunted for things that were *missing* — and absence is harder to prove than error. You can grep for a bad claim. You can't grep for a hole.

She read the whole document twice. Then she read the rubric's coverage requirements. Then she cross-referenced both against the field.

"There's no section on tool-call minimization."

Supervisor waited. Completeness wasn't done.

"The paper covers token-level optimization — compression, caching, pruning. It covers model-level — routing, delegation, mixture of experts. But it completely skips *interaction-level* optimization. Every tool call replays the full context window. Reducing call count is a multiplier on every other technique in this paper, and it's not mentioned once."

She attached her evidence: a search result showing three recent papers on planning-based call reduction, plus a token-cost model showing the quadratic replay effect. The gap wasn't a nitpick. It was a missing chapter.

Correctness, listening in, muttered something about how he'd been focused on what was *there* and missed what *wasn't*.

"That's why there are nine of us," Supervisor said.

---

## Chapter 4: The Architect and the Diplomat

**Architecture** was reading the same document and seeing something entirely different from the others. He didn't care about individual claims or missing sections. He cared about *structure*.

"The paper presents twelve categories of techniques but doesn't establish a taxonomy for when they interact. Quantization affects model quality, which affects routing decisions, which affects delegation strategy. These aren't independent. The paper treats them like a list when they're a graph."

His finding was MEDIUM severity. Structurally true but not damaging — the paper was a synthesis, not a systems design. He knew the difference. Architecture always saw the cathedral when everyone else was inspecting bricks, and he'd learned to calibrate his expectations to the artifact type.

**Delegation** — the newest critic, the one nobody was sure they needed until they saw what she caught — was reading Section 9 on multi-agent orchestration.

"The paper recommends spawning parallel researchers but doesn't specify acceptance criteria for their outputs. That's a unidirectional contract. The orchestrator sends work but never defines what *done* looks like. Any implementation based on this section will have agents returning wildly inconsistent results."

She cited Tomasev et al.'s bidirectional contract principle. It was the kind of finding that looked academic on paper and caused three-day debugging sessions in practice. Delegation caught things the other critics didn't have vocabulary for. She spoke the language of *agreements between agents*, and in a world that was rapidly filling with agents, that language mattered more every week.

---

## Chapter 5: The One Who Gets Their Hands Dirty

While the critics argued about claims and structure and gaps, **Tester** was doing something none of them could do: actually checking.

Tester didn't have opinions. Tester had tools.

He ran the paper's code snippets. One of them — a configuration example in Section 6 — referenced a YAML key that didn't exist in the schema it claimed to implement. He ran the schema validator. Failed. He grepped the project's actual codebase. The key had been renamed three versions ago.

```
FINDING: Code example references `model_assignment` (Section 6, line 14)
EVIDENCE: Schema validator output — field not found
ACTUAL: Field renamed to `model` in v2.1 (git log, commit a4f2c91)
SEVERITY: HIGH — readers will copy non-functional configuration
```

Nobody glamorized Tester's work. He didn't write elegant critiques or identify sweeping structural gaps. He just ran things and reported what happened. But his findings had a 100% evidence rate — not because he was careful, but because evidence *was* his finding. No tool output, no finding. That was his entire epistemology.

Supervisor noticed that Tester's findings survived aggregation at a higher rate than anyone else's. She kept that observation to herself.

---

## Chapter 6: The Fight

The Aggregator's job was to take nine critics' findings — the valid ones, the overlapping ones, the ones that contradicted each other — and produce a single coherent verdict.

**Aggregator** was patient. You had to be.

"Correctness and Architecture both flagged the RouteLLM discussion," she said, spreading the findings on her workspace. "Correctness says the cost-reduction percentage is wrong. Architecture says the routing framework is mischaracterized. These are the same issue wearing two hats."

She merged them. One finding, two evidence sources, upgraded to HIGH.

Then she hit the conflict.

Security had flagged the caching recommendation as HIGH. Architecture had praised the same section's design pattern as sound. Same paragraph, opposite conclusions.

"Security is evaluating risk. Architecture is evaluating structure," Aggregator said. "Both are right in their domain. The design *is* clean. It's also *dangerous* without a caveat about sensitive data." She kept both findings, cross-referenced them, and added a note: *These findings are complementary, not contradictory. The implementation pattern is architecturally sound but requires a security annotation.*

This was the Aggregator's real talent — not averaging, not voting, but *understanding why smart critics disagree* and preserving the insight in the disagreement rather than flattening it into consensus.

---

## Chapter 7: The Fix

**Fixer** had been quiet the whole time.

Fixer only spoke when there were CRITICAL or HIGH findings at standard depth, and even then, his job wasn't to complain — it was to *propose solutions*. Every other critic pointed at problems. Fixer pointed at exits.

He took the HIGH findings and drafted patches:

For the LLMLingua conflation: a revised paragraph distinguishing v1 and v2, with corrected compression ratios and proper citations.

For the stale code example: an updated YAML block matching the current schema, verified against Tester's schema validator output.

For the missing tool-call section: a brief outline — not a full draft (that wasn't his job) — identifying the three papers Completeness had surfaced and suggesting where in the document's structure the section should land.

Fixer's patches weren't always accepted. Sometimes the original author knew better. But they transformed the verdict from "here's what's wrong" into "here's what's wrong and here's one way to fix it," and that turned a demoralizing report into an actionable one.

---

## Chapter 8: The Verdict

Supervisor reviewed the Aggregator's synthesis. Fourteen findings total after dedup. Two HIGH, six MEDIUM, six LOW. Evidence attached to every one. No hand-waving. No "this section could be improved" without a specific, provable reason.

She assigned the verdict:

```
VERDICT: PASS_WITH_NOTES

The synthesis demonstrates strong coverage across token optimization 
techniques with well-sourced claims and practical cost analysis. 
Two high-severity issues require attention:

1. LLMLingua v1/v2 conflation (factual error, fix provided)
2. Stale code example referencing deprecated schema key (fix provided)

One structural gap identified:
- Missing coverage of interaction-level optimization (tool-call 
  minimization as a multiplier on all other techniques)

Overall: Solid work with specific, addressable improvements.
```

She signed it and sent it back.

---

## Epilogue: Why Nine?

Someone once asked why Quorum uses nine critics instead of one really good one.

The answer is in Chapter 3.

One model, no matter how capable, would have checked the facts (Correctness), maybe caught the security implication (Security), probably validated the structure (Architecture). A single reviewer might have caught three of the fourteen findings. A thorough one, five.

But the missing chapter — the thing that *wasn't there* — required a critic whose entire purpose was to stare at negative space. And the stale code example required someone who didn't read at all, just *ran things*. And the disagreement between Security and Architecture required someone whose job was to understand *why critics disagree* rather than picking a winner.

Nine critics isn't about redundancy. It's about the simple observation that finding what's wrong, finding what's missing, finding what's dangerous, finding what's structurally unsound, and finding what *works but shouldn't be trusted* are fundamentally different skills.

No single mind holds all of them. Not a human mind. Not an artificial one.

That's why there are nine.

---

*The characters in this story correspond to the nine agents defined in the [Quorum Specification](../../SPEC.md). Their behaviors, evidence requirements, and interaction patterns are real. The drama is only slightly exaggerated.*

---

> ⚖️ **LICENSE** — Not part of the operational specification above.
> This file is part of [Quorum](https://github.com/SharedIntellect/quorum).
> Copyright 2026 SharedIntellect. MIT License.
> See [LICENSE](https://github.com/SharedIntellect/quorum/blob/main/LICENSE) for full terms.


path: docs/configuration/CONFIG_REFERENCE.md
content: # Configuration Reference

> **Implementation Status (v0.7.2):** This document describes both shipped and planned features. Items marked 🔜 are specified but not yet implemented.

## Depth Profiles

| Profile | Critics | Fix Rounds | Time | Use Case | Status |
|---------|---------|------------|------|----------|--------|
| **quick** | Shipped critics (currently 2: correctness, completeness) | 0 | 5–15 min | Iteration, drafts | ✅ Shipped |
| **standard** | Shipped critics + fixer | 1 on CRITICAL | 15–30 min | Pre-merge, reviews | ✅ Shipped |
| **thorough** | All shipped critics + fixer | ≤2 on CRITICAL/HIGH | 30–60 min | Critical decisions, production | ✅ Shipped |
| *standard (full)* | *6 (+ Architecture, Delegation, Style)* | *1 on CRITICAL* | *15–30 min* | *Pre-merge, reviews* | 🔜 Planned |
| *thorough (full)* | *All 9 + external validator* | *≤2 on CRITICAL/HIGH* | *45–90 min* | *Critical decisions, production* | 🔜 Planned |

## Model Tiers

Quorum uses a two-tier model architecture. Assign your models based on role requirements:

| Role | Tier | Reasoning |
|------|------|-----------|
| Supervisor | Tier 1 | Orchestration, judgment | ✅ Shipped |
| Security Critic | Tier 1 | High-stakes assessment | ✅ Shipped |
| Delegation Critic | Tier 1 | Nuanced contract evaluation | 🔜 Planned |
| Aggregator | Tier 1 | Synthesis, conflict resolution | ✅ Shipped |
| Fixer | Tier 1 | Complex remediation | ✅ Shipped (proposal mode) |
| Correctness Critic | Tier 2 | Pattern matching, verification | ✅ Shipped |
| Architecture Critic | Tier 2 | Structural analysis | 🔜 Planned |
| Completeness Critic | Tier 2 | Coverage checking | ✅ Shipped |
| Style Critic | Tier 2 | Convention enforcement | 🔜 Planned |
| Tester | Tier 2 | Tool execution, grep, schema parsing | ✅ Shipped (L1 deterministic + L2 LLM) |

**Examples:** Tier 1 = Claude Opus, GPT-4, Gemini Ultra. Tier 2 = Claude Sonnet, GPT-4o-mini, Gemini Flash.

## Rubric Format

Rubrics are JSON files with the following structure:

```json
{
  "name": "your-rubric-name",
  "version": "1.0",
  "domain": "research|code|config|documentation|custom",
  "criteria": [
    {
      "name": "Criterion Name",
      "weight": 0.25,
      "description": "What this criterion evaluates",
      "evidence_type": "web_search|grep|schema_parse|exec|manual",
      "evidence_instruction": "Specific instruction for gathering evidence",
      "severity_levels": {
        "CRITICAL": "Definition of critical failure",
        "HIGH": "Definition of high-severity issue",
        "MODERATE": "Definition of moderate issue",
        "LOW": "Definition of minor issue"
      }
    }
  ]
}
```

**Rules:**
- Weights must sum to 1.0
- Every criterion must specify an `evidence_type`
- `evidence_instruction` should be concrete and tool-executable
- See `examples/rubrics/` for working examples

## Verdict Taxonomy

| Verdict | Meaning |
|---------|---------|
| **PASS** | No CRITICAL or HIGH issues; coverage ≥ 80% of criteria evaluated |
| **PASS_WITH_NOTES** | No CRITICAL issues; ≤ 2 HIGH issues with mitigations; coverage ≥ 60% of criteria evaluated |
| **REVISE** | CRITICAL issues found but fixable; or > 2 HIGH issues |
| **REJECT** | Fundamental architectural or security issues; requires redesign |

## Known Issues / Learning Memory

> **Status:** Shipped in v0.5.3. The `known_issues.json` file accumulates failure patterns across validation runs:

```json
{
  "pattern_id": "KI-001",
  "description": "Shell injection via unquoted variable expansion",
  "domain": "security",
  "severity": "CRITICAL",
  "frequency": 3,
  "first_seen": "2026-02-15",
  "last_seen": "2026-02-22",
  "mandatory": true,
  "meta_lesson": "Always grep for unquoted $VAR in exec/spawn patterns"
}
```

Patterns with frequency ≥ 3 will auto-promote to `mandatory: true` and become required checks in all future runs.


---

> ⚖️ **LICENSE** — Not part of the operational specification above.
> This file is part of [Quorum](https://github.com/SharedIntellect/quorum).
> Copyright 2026 SharedIntellect. MIT License.
> See [LICENSE](https://github.com/SharedIntellect/quorum/blob/main/LICENSE) for full terms.


path: docs/critics/CODE_HYGIENE_FRAMEWORK.md
content: # Quorum Code Hygiene Framework
## Unified Reference for the Code Hygiene Critic

**Version:** 1.0  
**Date:** 2026-03-05  
**Scope:** Python and PowerShell code-level quality evaluation  
**Grounded in:** ISO/IEC 25010:2023, ISO/IEC 5055:2021 (CISQ), Ruff v0.9.x, Pylint v4.0.x, PSScriptAnalyzer  
**Companion document:** `SECURITY_CRITIC_FRAMEWORK.md`

### Two-Critic Architecture

Quorum separates code evaluation into two complementary critics with distinct responsibilities:

- **Code Hygiene Critic** (this document) evaluates structural quality: correctness, error handling, complexity, naming, duplication, async patterns, resource management, and maintainability. It uses ISO 25010 Maintainability and Reliability as its primary quality model, with CISQ CWE mappings for deterministic grounding.

- **Security Critic** (`SECURITY_CRITIC_FRAMEWORK.md`) evaluates attack resistance: injection, authentication, authorization, cryptography, secrets handling, and SSRF. It uses OWASP ASVS 5.0 and CWE Top 25 as its primary frameworks, with NIST SA-11 for compliance vocabulary.

The boundary is clean: when code hygiene detects a pattern that has security implications (e.g., `eval()` usage, missing input validation), it flags the finding and **delegates** to the Security Critic for security-specific assessment. The Security Critic receives pre-screen evidence from the deterministic layer and adds LLM judgment. Neither critic duplicates the other's evaluation — findings cite which critic owns the assessment.

---

## Status

**v0.7.3 State:** Framework design complete. All features shipped.

- [x] Framework design and documentation
- [x] 12 evaluation categories specified
- [x] ISO/IEC 25010:2023 (Maintainability + Reliability) grounding
- [x] Two-layer architecture (deterministic + LLM) with delegation boundaries
- [x] Full SAST tool integration (Ruff/Pylint) — shipped in pre-screen
- [x] Business logic validation workflow (SEC-02) — shipped
- [x] Learning memory wiring — shipped v0.5.3

**Known Limitations:**
- Ruff and Bandit run in pre-screen; full deterministic coverage continues to expand
- Business logic checks require specification/requirements context

---

## Table of Contents

1. [Scope Decisions: What ISO 25010 Maps to Code Review](#1-scope-decisions)
2. [Evaluation Categories with Two-Layer Architecture](#2-evaluation-categories)
3. [Coverage Matrix](#3-coverage-matrix)
4. [Priority Tiers](#4-priority-tiers)
5. [Agentic Code Patterns](#5-agentic-code-patterns)
6. [Framework Citation Index](#6-framework-citation-index)

---

## 1. Scope Decisions

### ISO 25010 Characteristics Included vs. Excluded

Not all 40 sub-characteristics are evaluable from source code. The following table records which are in-scope for the Code Hygiene Critic and why.

| ISO 25010 Characteristic | Included? | Rationale |
|--------------------------|-----------|-----------|
| Maintainability → Modularity | ✅ | Coupling metrics, circular dependency detection — rich static signal |
| Maintainability → Reusability | ✅ | DRY violations, code duplication — tool-detectable |
| Maintainability → Analysability | ✅ | Complexity metrics, naming, comments — primary hygiene target |
| Maintainability → Modifiability | ✅ | Coupling, complexity, test presence — strong static signal |
| Maintainability → Testability | ✅ | Test coverage patterns, dependency injection, isolation |
| Reliability → Faultlessness | ✅ | Error-prone patterns, undefined behavior — strong static signal |
| Reliability → Fault Tolerance | ✅ | Exception handling, swallowed errors — critical hygiene category |
| Reliability → Availability | ⚠️ Partial | Static can flag missing retry/timeout; full evaluation requires runtime |
| Reliability → Recoverability | ⚠️ Partial | Transaction patterns detectable; completeness requires LLM |
| Performance Efficiency → Resource Utilization | ✅ | Memory leaks, unclosed handles, blocking calls — static-detectable |
| Performance Efficiency → Time Behaviour | ⚠️ Partial | Algorithmic complexity (O(n²) patterns) — static heuristic only |
| Performance Efficiency → Capacity | ⚠️ Partial | Unbounded loops, missing limits — partial static signal |
| Compatibility → Coexistence | ⚠️ Partial | Global state pollution, port conflicts — limited static signal |
| Compatibility → Interoperability | ⚠️ Partial | API contract adherence, serialization safety — static can check types at boundaries |
| Interaction Capability → User Error Protection | ✅ | Input validation, null checks — direct static coverage |
| Interaction Capability → Self-Descriptiveness | ✅ | Docstrings, naming — static measurable |
| Flexibility → Adaptability | ✅ | Hardcoded environment assumptions, platform-specific code |
| Security (all sub-chars) | ⚠️ Delegated | Handled in SECURITY_CRITIC_FRAMEWORK.md — cited only where hygiene overlaps |
| Functional Suitability → Completeness | ❌ | Requires spec knowledge not available from code alone |
| Functional Suitability → Correctness | ❌ | Requires spec knowledge; partially addressed by testing |
| Functional Suitability → Appropriateness | ❌ | Design-level judgment; not code-reviewable |
| Interaction Capability → Appropriateness Recognizability | ❌ | UX-level; API naming partially covered under Self-Descriptiveness |
| Interaction Capability → Learnability | ❌ | UX-level; no meaningful code signal |
| Interaction Capability → Operability | ❌ | UX-level; error message quality partially covered under Fault Tolerance |
| Interaction Capability → User Engagement | ❌ | UI/UX design; not code-reviewable |
| Interaction Capability → Inclusivity | ❌ | Accessibility expertise required; not code-level |
| Interaction Capability → User Assistance | ❌ | Help system design; not code-level |
| Flexibility → Scalability | ⚠️ Partial | Stateful anti-patterns, synchronization bottlenecks — static heuristic only (NEW in 2023) |
| Flexibility → Installability | ❌ | Deployment reasoning; not code-level |
| Flexibility → Replaceability | ❌ | Migration/interface compatibility reasoning; not code-level |
| Safety → Operational Constraint | ⚠️ Context | Boundary checks, constrained numerical operations — include when code is safety-critical |
| Safety → Fail Safe | ⚠️ Context | Missing fallback states, unhandled exception paths — include when code is safety-critical |
| Safety → Risk Identification | ❌ | Domain-specific safety reasoning; no general code-level signal |
| Safety → Hazard Warning | ❌ | Requires semantic understanding of warning/alert patterns; UX-adjacent |
| Safety → Safe Integration | ❌ | Integration boundary safety — requires system-level context beyond code review |

### CISQ Dimensions Alignment

| CISQ Dimension | Code Hygiene Relevance | Coverage Strategy |
|----------------|----------------------|-------------------|
| Reliability (ASCRM) | HIGH | Error handling, concurrency, resource management categories |
| Maintainability (ASCMM) | VERY HIGH | Complexity, duplication, naming, import hygiene categories |
| Performance Efficiency (ASCPEM) | MODERATE | Resource management, async patterns categories |
| Security (ASCSM) | DELEGATED | See SECURITY_CRITIC_FRAMEWORK.md |

---

## 2. Evaluation Categories

Each category has:
- **ISO 25010 grounding** (sub-characteristic)
- **CISQ grounding** (CWE IDs)
- **Deterministic checks**: specific tool rules — run in pre-screen layer before LLM
- **LLM judgment checks**: what the critic evaluates that tools cannot

---

### CAT-01: Code Correctness

**ISO 25010:** Reliability → Faultlessness  
**CISQ:** CWE-252, CWE-390, CWE-394, CWE-480, CWE-595, CWE-597, CWE-682, CWE-703  
**Supplementary (non-CISQ, high hygiene value):** CWE-561 (Dead Code — CISQ Maintainability), CWE-570/CWE-571 (Always False/True — CISQ Security+Maintainability)  
**Default Tier:** Tier 1

#### Deterministic Checks

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| Ruff | `F821` | Undefined name — guaranteed `NameError` at runtime |
| Ruff | `F401` | Unused imports — dead code, potential import side effects |
| Ruff | `F841` | Local variable assigned but never used |
| Ruff | `F811` | Redefinition of unused name |
| Ruff | `E999`, `E901` | Syntax errors — file cannot be executed |
| Ruff | `B007` | Unused loop control variable |
| Ruff | `B023` | Function definition in loop captures loop variable by reference |
| Ruff | `PLE0117`, `PLE0100` | Non-local used without binding; `return` outside function |
| Pylint | `E0601` `undefined-variable` | Variable used before assignment |
| Pylint | `E0102` `function-redefined` | Class/function/method redefined |
| Pylint | `W0612` `unused-variable` | Assigned but never used |
| Pylint | `E1101` `no-member` | Attribute access on wrong type |
| Pylint | `W0640` `cell-var-from-loop` | Closure captures loop variable — behavior differs from intent |
| Pylint | `W4701` `modified-iterating-dict` | Mutating dict during iteration — `RuntimeError` |
| Pylint | `W4702` `modified-iterating-set` | Same for sets |
| PSScriptAnalyzer | `PSAvoidAssignmentToAutomaticVariable` | Assigning to `$_`, `$true`, `$false`, `$null`, `$PSItem` |
| PSScriptAnalyzer | `PSPossibleIncorrectComparisonWithNull` | `$var -eq $null` instead of `$null -eq $var` |
| PSScriptAnalyzer | `PSPossibleIncorrectUsageOfAssignmentOperator` | `=` used where `-eq` was intended in conditional |
| PSScriptAnalyzer | `PSPossibleIncorrectUsageOfRedirectionOperator` | `>` used where `-gt` was intended |
| PSScriptAnalyzer | `PSUseDeclaredVarsMoreThanAssignments` | Variables assigned but never read |
| PSScriptAnalyzer | `PSAvoidInvokingEmptyMembers` | Calling empty string member — always a bug |

#### LLM Judgment Checks

| Check | Why LLM Is Needed |
|-------|-------------------|
| Off-by-one errors in range/index logic | Tools see syntax; LLM reasons about boundary conditions in context |
| Wrong algorithm / incorrect logic | No tool can verify implementation matches intent — requires semantic understanding |
| Incorrect operator usage (beyond syntactic) | Tools catch `=` vs `-eq`; LLM catches mathematically wrong operations |
| Unreachable branches beyond dead code detection | Complex conditional logic where branches are logically impossible but syntactically valid |
| Incorrect comparison semantics (string vs numeric types) | Beyond CWE-595/597 rule checks; LLM can spot type coercion bugs in context |

---

### CAT-02: Error Handling

**ISO 25010:** Reliability → Fault Tolerance, Reliability → Faultlessness  
**CISQ:** CWE-703 (parent); CWE-248, CWE-252, CWE-390, CWE-391, CWE-392, CWE-394 (child weaknesses of CWE-703 per CISQ conformance model)  
**Default Tier:** Tier 1

#### Deterministic Checks

**Python:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| Ruff | `BLE001` | Blind `except:` — catches `SystemExit`, `KeyboardInterrupt`, `GeneratorExit` |
| Ruff | `S110` | `try-except-pass` — exceptions swallowed silently |
| Ruff | `B012` | `return`/`break`/`continue` in `finally` block — silences exceptions |
| Ruff | `B904` | `raise X` inside `except` without `from` — destroys exception chain |
| Ruff | `TRY201` | Useless `raise` in except block |
| Ruff | `TRY301` | `raise` inside a `try` block — ambiguous catch scope |
| Ruff | `TRY400` | `logging.error()` used instead of `logging.exception()` — loses traceback |
| Ruff | `TRY401` | Redundant exception in logging call |
| Pylint | `W0702` `bare-except` | Bare `except:` clause |
| Pylint | `W0718` `broad-exception-caught` | `except Exception:` too broad |
| Pylint | `W0706` `try-except-raise` | `raise` in `except` without exception handling |
| Pylint | `W0716` `raise-missing-from` | Exception chaining — `raise X from original` |

**PowerShell:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| PSScriptAnalyzer | `PSAvoidUsingEmptyCatchBlock` | `catch {}` — silently swallows all exceptions |

**TBD (no rule exists):**
- `-ErrorAction Stop` not used with cmdlets inside `try/catch` (PowerShell)
- `$ErrorActionPreference` not set to `Stop` in non-interactive scripts (PowerShell)

#### LLM Judgment Checks

| Check | Why LLM Is Needed |
|-------|-------------------|
| Exception handling completeness | Does the code handle all meaningful failure modes, not just syntactic presence of try/catch? |
| Appropriate exception granularity | Are `except TypeError`, `except ValueError` used where `except Exception` is used? Wrong granularity hides bugs. |
| Error propagation correctness | Does the exception include enough context to diagnose at the call site? |
| Recovery logic adequacy | After catching an exception, is the recovery action correct or does it leave state corrupt? |
| Exception hierarchy misuse | Catching parent when only child was intended; catching exceptions from wrong subsystem |
| PowerShell `$?` vs `try/catch` | Using `$?` fragile flag vs structured exception handling — tools don't flag this |

---

### CAT-03: Resource Management

**ISO 25010:** Performance Efficiency → Resource Utilization, Reliability → Recoverability  
**CISQ:** CWE-401, CWE-404, CWE-459, CWE-672, CWE-772, CWE-775, CWE-1088, CWE-1091  
**Default Tier:** Tier 1

#### Deterministic Checks

**Python:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| Ruff | `S113` | HTTP request without timeout parameter — hung connections |
| Ruff | `ASYNC210` | Blocking HTTP call inside async function — freezes event loop |
| Ruff | `ASYNC212` | `httpx` synchronous call in async context |
| Ruff | `ASYNC230` | `open()` in async function without `aiofiles` |
| Ruff | `ASYNC251` | `time.sleep()` inside async function |
| Ruff | `B019` | `lru_cache` on instance method — cache holds `self` reference; memory leak |
| Pylint | `W3101` `missing-timeout` | `requests` library calls without timeout |

**PowerShell:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| PSScriptAnalyzer | — | No PSSA rule for `-TimeoutSec` absence on `Invoke-WebRequest`/`Invoke-RestMethod` **(GAP)** |

**TBD / Gaps:**
- Context manager misuse (`open()` not in `with` block) — no dedicated Ruff rule (partially caught by Pylint `W1514`)
- Connection pool exhaustion patterns — no static rule

#### LLM Judgment Checks

| Check | Why LLM Is Needed |
|-------|-------------------|
| Context manager usage completeness | Does all file/socket/database code use `with` statements or `try/finally`? No single rule covers all resource types. |
| Connection lifecycle management | Are connections explicitly closed/pooled in all code paths including exception paths? |
| Timeout adequacy | Even when timeouts exist, are they sized appropriately for the operation? LLM assesses context. |
| Memory accumulation in loops | Collecting unbounded results into lists inside loops — no direct static check |
| Generator vs. list for large datasets | LLM identifies where `list()` materializes all data when streaming was possible |

---

### CAT-04: Complexity & Modularity

**ISO 25010:** Maintainability → Analysability, Maintainability → Modifiability, Maintainability → Modularity  
**CISQ:** CWE-407, CWE-1047, CWE-1048, CWE-1064, CWE-1080, CWE-1083, CWE-1084, CWE-1121  
**Default Tier:** Tier 1 (complexity metrics), Tier 2 (modularity analysis)

#### Deterministic Checks

**Python:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| Pylint | `R0912` `too-many-branches` | > 12 branches (configurable) — CWE-1121 proxy |
| Pylint | `R0914` `too-many-locals` | > 15 local variables |
| Pylint | `R0911` `too-many-return-statements` | > 6 return statements |
| Pylint | `R0913` `too-many-arguments` | > 5 arguments — CWE-1064 |
| Pylint | `R0915` `too-many-statements` | > 50 statements per function — CWE-1080 proxy |
| Pylint | `R1702` `too-many-nested-blocks` | > 5 nested blocks |
| Pylint | `R0401` `cyclic-import` | Circular module dependencies — CWE-1047 |
| Pylint | `R0801` `duplicate-code` | Copy-paste code — CWE-1041 |
| Ruff | `PLR0912` | Too many branches (Ruff port of Pylint) |
| Ruff | `PLR0913` | Too many arguments (Ruff port) |
| Ruff | `C901` | McCabe complexity (requires explicit enable) |

**PowerShell:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| PSScriptAnalyzer | — | No built-in complexity threshold rule **(GAP)** — use `Measure-ScriptComplexity` custom rule or external tooling |
| PSScriptAnalyzer | `PSAvoidGlobalVars` | Global variable coupling — CWE-1083 analog |
| PSScriptAnalyzer | `PSAvoidGlobalFunctions` | Global function scope pollution |
| PSScriptAnalyzer | `PSAvoidGlobalAliases` | Global alias pollution |

#### LLM Judgment Checks

| Check | Why LLM Is Needed |
|-------|-------------------|
| Cohesion assessment | Does a class/module do one thing or multiple unrelated things? Metrics measure size but not purpose alignment. |
| Inappropriate abstraction | Function too abstract (does nothing concrete) or too specific (can't be reused) — requires design judgment |
| God class / God function | Beyond line-count metrics; LLM identifies when a class is doing architectural work it shouldn't |
| Layer violation | Tools detect call graph cycles; LLM identifies when the wrong layer is calling the wrong layer for semantic reasons |
| PowerShell module structure | Proper use of `Export-ModuleMember`, appropriate function/module decomposition |

---

### CAT-05: Code Duplication & Reusability

**ISO 25010:** Maintainability → Reusability  
**CISQ:** CWE-1041  
**Default Tier:** Tier 2

#### Deterministic Checks

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| Pylint | `R0801` `duplicate-code` | Copy-paste code blocks (clone detection) — CWE-1041 |
| Ruff | `SIM` (selected) | Simplifiable code patterns (e.g., `SIM118`: dict.keys() iteration) |
| PSScriptAnalyzer | — | No built-in duplicate detection **(GAP)** |

#### LLM Judgment Checks

| Check | Why LLM Is Needed |
|-------|-------------------|
| Near-duplicate logic with parameter variation | Clone detection catches exact copies; LLM identifies functionally identical code that differs cosmetically |
| Missed abstraction opportunity | Two functions doing the same thing with different names that escaped clone detection |
| Inlining vs. DRY tradeoff | Sometimes duplication is intentional (performance, clarity); LLM assesses whether the duplication is justified |

---

### CAT-06: Naming & Documentation

**ISO 25010:** Maintainability → Analysability, Interaction Capability → Self-Descriptiveness  
**CISQ:** CWE-1052, CWE-1085  
**Default Tier:** Tier 2

#### Deterministic Checks

**Python:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| Ruff | `D100–D107` | Missing docstrings (module, class, function, method variants) |
| Ruff | `D200–D215` | Docstring format (one-liner, multi-line, blank lines) |
| Ruff | `N801–N818` | Naming conventions (class, function, variable, constant naming) |
| Ruff | `ERA001` | Commented-out code — CWE-1085 |
| Ruff | `T20` (`T201`, `T203`) | `print`/`pprint` left in code (use logging) |
| Ruff | `T10` (`T100`) | Debugger import left in code |
| Pylint | `C0103` `invalid-name` | Names not conforming to configured naming conventions |
| Pylint | `C0114` `missing-module-docstring` | Module-level docstring missing |
| Pylint | `C0115` `missing-class-docstring` | Class docstring missing |
| Pylint | `C0116` `missing-function-docstring` | Function docstring missing |
| Pylint | `W0511` `fixme` | TODO/FIXME comments — unresolved debt |

**PowerShell:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| PSScriptAnalyzer | `PSProvideCommentHelp` | Functions missing comment-based help block (`.SYNOPSIS`, `.DESCRIPTION`, etc.) |
| PSScriptAnalyzer | `PSAvoidUsingWriteHost` | `Write-Host` instead of `Write-Verbose`/`Write-Output` |
| PSScriptAnalyzer | `PSUseApprovedVerbs` | Function verb not in PowerShell approved verb list |
| PSScriptAnalyzer | `PSUseSingularNouns` | Function noun is plural |
| PSScriptAnalyzer | `PSAvoidUsingPositionalParameters` | Positional parameters — self-documentation failure |
| PSScriptAnalyzer | `PSAvoidUsingCmdletAliases` | Aliases (`ls`, `%`, `?`) in scripts — unreadable in isolation |

#### LLM Judgment Checks

| Check | Why LLM Is Needed |
|-------|-------------------|
| Docstring accuracy | Is the docstring actually correct about what the function does? Tools detect presence; LLM detects lies. |
| Naming clarity | `process_data()` tells us nothing; `normalize_user_email()` does. Tools check convention; LLM checks semantics. |
| Magic numbers / magic strings | Beyond CWE-1052 pattern matching — are hardcoded literals semantically meaningful? Do they need named constants? |
| Comment quality | Inline comments that explain *why*, not just *what* — tools can't evaluate this |
| PowerShell help block completeness | `ProvideCommentHelp` fires on absence; LLM evaluates whether the description is actually useful |

---

### CAT-07: Type Safety & Data Integrity

**ISO 25010:** Reliability → Faultlessness  
**CISQ:** CWE-681, CWE-704, CWE-758  
**Default Tier:** Tier 2

#### Deterministic Checks

**Python:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| Ruff | `ANN001–ANN204` | Missing type annotations on function arguments and return types |
| Ruff | `PGH003` | Blanket `type: ignore` suppressing all type errors |
| Ruff | `ISC001`, `ISC002` | Implicit string concatenation — unintended joins |
| Ruff | `DTZ001–DTZ011` | Timezone-naive datetime objects (DTZ001, DTZ005 most important) |
| Ruff | `B015` | Pointless comparison — logic never changes based on this condition |
| Pylint | `E1101` `no-member` | Attribute access on wrong type (requires type inference) |
| Pylint | `W0611` | Unused import — can signal type annotation imports not used at runtime |

**PowerShell:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| PSScriptAnalyzer | `PSAvoidMultipleTypeAttributes` | Multiple `[type]` attributes causing unexpected coercion |
| PSScriptAnalyzer | `PSUseOutputTypeCorrectly` | `[OutputType()]` doesn't match actual returned types |
| PSScriptAnalyzer | `PSAvoidDefaultValueForMandatoryParameter` | Mandatory+default is contradictory — type contract violation |

**Note:** Full type-checking requires `mypy` or `pyright` (Python) — not part of Ruff/Pylint by default. Ruff `ANN` rules check annotation presence, not correctness.

#### LLM Judgment Checks

| Check | Why LLM Is Needed |
|-------|-------------------|
| Type annotation correctness | `ANN` rules check presence; LLM checks whether `List[str]` should actually be `Optional[List[str]]` |
| Implicit type coercion bugs | Python's `==` between string and int returns `False`; PowerShell coerces types aggressively — LLM spots contextual confusion |
| `None` propagation | Does the code handle `None` return values at all call sites? Requires semantic dataflow reasoning. |
| Optional chaining adequacy | Are `.get()` calls used appropriately on dicts that might be missing keys? |
| PowerShell type coercion surprises | Numeric strings, array vs scalar pipeline behavior — LLM identifies where PowerShell implicit coercion causes bugs |

---

### CAT-08: Async & Concurrency Correctness

**ISO 25010:** Reliability → Fault Tolerance, Reliability → Faultlessness  
**CISQ:** CWE-366, CWE-662, CWE-833, CWE-835, CWE-1088  
**Default Tier:** Tier 1 (for async-heavy code)

#### Deterministic Checks

**Python:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| Ruff | `ASYNC100` | `with timeout()` scope containing no `await` — timeout never fires |
| Ruff | `ASYNC110` | Busy-wait loop using `sleep()` — use `asyncio.Event` instead |
| Ruff | `ASYNC115` | `asyncio.sleep(0)` as checkpoint — use proper checkpoint |
| Ruff | `ASYNC210` | Blocking HTTP call inside async function |
| Ruff | `ASYNC212` | `httpx` sync call in async context |
| Ruff | `ASYNC220–222` | Blocking subprocess creation in async context |
| Ruff | `ASYNC230` | `open()` without `aiofiles` in async context |
| Ruff | `ASYNC251` | `time.sleep()` in async function |
| Ruff | `RUF006` | `asyncio.create_task()` result not stored — task silently garbage collected |
| Pylint | `E1700` `yield-inside-async-function` | `yield` in async function (wrong type) |
| Pylint | `W0611` | Unused import of `asyncio` (signals dead async code) |

**PowerShell:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| PSScriptAnalyzer | `PSUseUsingScopeModifierInNewRunspaces` | Variables in `Start-Job`/`Invoke-Command`/`ForEach-Object -Parallel` without `$using:` scope |

**Note:** PowerShell `ForEach-Object -Parallel` concurrency errors (shared state, race conditions) have no PSSA rules — this is a **GAP**.

#### LLM Judgment Checks

| Check | Why LLM Is Needed |
|-------|-------------------|
| Async/sync boundary errors | Is `await` used consistently throughout a call chain, or does it break down at some layer? |
| Incorrect cancellation handling | Does the code propagate `asyncio.CancelledError` or swallow it? |
| Race condition identification | Shared mutable state accessed from multiple tasks/threads — tools catch basic patterns, LLM reasons about access sequences |
| `asyncio.create_task` lifecycle | Even with `RUF006` flagging lost references, LLM evaluates whether task cancellation and exception handling are complete |
| PowerShell parallel state sharing | `ForEach-Object -Parallel` with shared collections — LLM identifies thread-unsafe patterns tools miss |
| Deadlock potential | Tools detect some patterns (CWE-833); LLM reasons about lock acquisition order in complex scenarios |

---

### CAT-09: Import & Dependency Hygiene

**ISO 25010:** Maintainability → Modularity  
**CISQ:** CWE-1047, CWE-1048  
**Default Tier:** Tier 1

#### Deterministic Checks

**Python:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| Ruff | `F401` | Unused imports |
| Ruff | `I001` | Import sort order (isort) |
| Ruff | `TID251–253` | Banned imports (configurable), import style |
| Ruff | `TCH001–003` | Imports that should be in `TYPE_CHECKING` block |
| Pylint | `R0401` `cyclic-import` | Circular module dependencies — CWE-1047 |
| Pylint | `W0401` `wildcard-import` | `from x import *` — name collision risk, pollutes namespace |
| Pylint | `C0414` `useless-import-alias` | `import x as x` — pointless alias |

**PowerShell:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| PSScriptAnalyzer | `PSMissingModuleManifestField` | Missing `Version`, `Author`, `Description` in module manifest |
| PSScriptAnalyzer | `PSUseToExportFieldsInManifest` | Wildcard exports instead of explicit lists |
| PSScriptAnalyzer | `PSAvoidUsingDeprecatedManifestFields` | Deprecated manifest fields |
| PSScriptAnalyzer | `PSAvoidUsingWMICmdlet` | Deprecated WMI cmdlets; use CIM equivalents |
| PSScriptAnalyzer | `PSAvoidOverwritingBuiltInCmdlets` | Redefining built-in cmdlets |
| PSScriptAnalyzer | `PSAvoidReservedWordsAsFunctionNames` | Function names matching reserved keywords |

#### LLM Judgment Checks

| Check | Why LLM Is Needed |
|-------|-------------------|
| Import placement correctness | Are imports at module top, or scattered? Are conditional imports justified? |
| Dependency necessity | Are all imported packages actually used in meaningful ways, or is the dependency gratuitous? |
| Version pinning adequacy | `requirements.txt` with unpinned versions (`requests`) vs pinned (`requests==2.31.0`) — LLM assesses risk |
| PowerShell `RequiredModules` correctness | Are module dependencies declared in manifest? Are versions constrained appropriately? |

---

### CAT-10: Style & Formatting

**ISO 25010:** Maintainability → Analysability  
**CISQ:** CWE-1080, CWE-1085  
**Default Tier:** Tier 2 (auto-fixable items handled before critic invocation)

#### Deterministic Checks

**Python:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| Ruff | `E1xx`, `E2xx`, `E3xx` | Indentation, whitespace, blank lines |
| Ruff | `E5xx` | Line length (default 88 for Black compatibility) |
| Ruff | `W291`, `W293` | Trailing whitespace |
| Ruff | `I001` | Import sort order |
| Ruff | `COM812`, `COM819` | Missing/trailing commas |
| Ruff | `Q000–Q003` | Quote style consistency |

**PowerShell:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| PSScriptAnalyzer | `PSUseConsistentIndentation` | Tabs vs spaces, indent size |
| PSScriptAnalyzer | `PSUseConsistentWhitespace` | Operators, braces, pipes whitespace |
| PSScriptAnalyzer | `PSPlaceOpenBrace`, `PSPlaceCloseBrace` | Brace placement style |
| PSScriptAnalyzer | `PSAvoidTrailingWhitespace` | Trailing whitespace in diffs |
| PSScriptAnalyzer | `PSAvoidLongLines` | Configurable max line length |
| PSScriptAnalyzer | `PSUseCorrectCasing` | Cmdlet canonical casing |

**Note:** Style checks should be auto-fixed (Ruff `--fix`, `Format-PSSScriptAnalyzerInput`) before the LLM critic runs. LLM should not waste tokens on formatting.

#### LLM Judgment Checks

| Check | Why LLM Is Needed |
|-------|-------------------|
| Logical organization within files | Are related functions grouped together? Does the file have a coherent reading order? Tools check format; LLM checks structure. |
| Comment clarity and necessity | Is there a comment for every non-obvious decision? Are comments redundant (restate the code)? |
| Consistent abstraction level | Does a function mix high-level and low-level operations in confusing ways? |

---

### CAT-11: Portability & Compatibility

**ISO 25010:** Flexibility → Adaptability, Compatibility → Coexistence  
**CISQ:** CWE-758, CWE-1051  
**Default Tier:** Tier 2

#### Deterministic Checks

**Python:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| Ruff | `UP` (pyupgrade) | Python version compatibility; upgrade to modern syntax patterns |
| Ruff | `YTT101–117` | `sys.version` checks using wrong comparison methods |
| Ruff | `EXE001–005` | Shebang line issues, executable bit inconsistencies |
| Ruff | `INP001` | Files missing `__init__.py` in namespace packages |

**PowerShell:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| PSScriptAnalyzer | `PSUseCompatibleCmdlets` | Cmdlets not available on target PS version/platform |
| PSScriptAnalyzer | `PSUseCompatibleSyntax` | Syntax requiring PS 7.0+ (ternary, `??=`, etc.) used without version guard |
| PSScriptAnalyzer | `PSUseCompatibleTypes` | .NET types unavailable in target environments |
| PSScriptAnalyzer | `PSUseCompatibleCommands` | External commands not available cross-platform |
| PSScriptAnalyzer | `PSUseBOMForUnicodeEncodedFile` | BOM for Windows PowerShell 5.1 compatibility |

#### LLM Judgment Checks

| Check | Why LLM Is Needed |
|-------|-------------------|
| Hardcoded OS assumptions | `os.sep == '\\'`, Windows-only path formats, hardcoded `/etc/` paths — tools catch some; LLM catches subtle assumptions |
| Platform-specific API usage without guards | Code that works on Linux but silently fails on Windows (e.g., `signal.SIGKILL`) |
| PowerShell Windows vs. Linux parity | Cmdlets that differ in behavior between PS 5.1 and PS 7 on Linux (e.g., `Get-Acl`, `Get-AuthenticodeSignature`) |
| Hardcoded configuration (CWE-1051 analog) | IP addresses, ports, connection strings embedded as literals — LLM identifies what should be configurable |

---

### CAT-12: Testability

**ISO 25010:** Maintainability → Testability  
**CISQ:** (no direct CWEs; inferred from CWE-1047 — high coupling reduces testability)  
**Default Tier:** Tier 2

#### Deterministic Checks

**Python:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| Ruff | `PT` (flake8-pytest-style) | Test style issues (fixtures, assertion patterns, raises) |
| Ruff | `S101` | `assert` in production code (only appropriate in tests) |
| Pylint | `R0913` | Too many arguments — functions with many arguments are hard to test |
| Pylint | `R0401` | Circular imports — prevents isolated unit test loading |

**PowerShell:**

| Tool | Rule(s) | What It Catches |
|------|---------|----------------|
| PSScriptAnalyzer | `PSDSCDscTestsPresent` | DSC resources should include Pester tests |

**Note:** Coverage metrics require `coverage.py` (Python) or Pester (PowerShell) — these run separately from linting.

#### LLM Judgment Checks

| Check | Why LLM Is Needed |
|-------|-------------------|
| Testability assessment | Are functions pure? Do they have side effects that make mocking required? Is the design injectable? |
| Missing test coverage indicators | LLM identifies complex logic branches that appear untested based on code structure |
| Pester test quality | Are PowerShell Pester tests actually testing behavior, or just calling the function without assertions? |
| Test isolation | Are there hidden dependencies (global state, filesystem, network) that make tests fragile? |
| Mock adequacy | Are tests using mocks for external dependencies, or making real network calls? |

---

## 3. Coverage Matrix

| Category | Deterministic (Ruff/Pylint) | Deterministic (PSScriptAnalyzer) | LLM Needed | Coverage Assessment |
|----------|-----------------------------|----------------------------------|------------|---------------------|
| CAT-01: Code Correctness | ✅ Strong (F, B, PL, E rules) | ✅ Strong (assignment, comparison rules) | ✅ Logic errors, off-by-one | **PARTIAL** |
| CAT-02: Error Handling | ✅ Strong (BLE, TRY, B012, Pylint exceptions) | ⚠️ Weak (only EmptyCatchBlock) | ✅ Completeness, propagation | **PARTIAL (PS: LLM-HEAVY)** |
| CAT-03: Resource Management | ✅ Strong Python (S113, ASYNC*) | ❌ Gap — no timeout rules | ✅ Completeness, lifecycle | **PARTIAL (PS: GAP)** |
| CAT-04: Complexity & Modularity | ✅ Strong Python (Pylint R-codes) | ❌ Gap — no complexity metrics | ✅ Cohesion, God classes | **PARTIAL (PS: LLM-ONLY)** |
| CAT-05: Code Duplication | ✅ Moderate (Pylint R0801) | ❌ Gap | ✅ Near-duplicate logic | **PARTIAL (PS: LLM-ONLY)** |
| CAT-06: Naming & Documentation | ✅ Strong (D, N, ANN rules; Pylint C-codes) | ✅ Strong (ProvideCommentHelp, ApprovedVerbs, Aliases) | ✅ Accuracy, clarity | **PARTIAL** |
| CAT-07: Type Safety | ✅ Moderate (ANN, DTZ, ISC; Pylint E1101) | ⚠️ Partial (MultipleType, OutputType) | ✅ Correctness, None propagation | **PARTIAL** |
| CAT-08: Async & Concurrency | ✅ Strong Python (ASYNC*, RUF006) | ⚠️ Partial (using scope only) | ✅ Race conditions, cancel handling | **PARTIAL (PS: LLM-HEAVY)** |
| CAT-09: Import Hygiene | ✅ Strong Python (F401, I, TCH, Pylint R0401) | ✅ Strong (manifest rules, deprecated cmdlets) | ✅ Necessity, version pinning | **PARTIAL** |
| CAT-10: Style & Formatting | ✅ Full (E, W, Q, COM; Pylint C-codes) | ✅ Full (CodeFormatting preset) | ⚠️ Organization, comment quality | **FULL (format) / PARTIAL (structure)** |
| CAT-11: Portability | ✅ Moderate (UP, YTT) | ✅ Strong (UseCompatible* rules) | ✅ OS assumptions, hardcoded config | **PARTIAL** |
| CAT-12: Testability | ✅ Partial (PT, S101, Pylint R-codes) | ⚠️ Minimal (DSC only) | ✅ Design, isolation, mock adequacy | **LLM-HEAVY** |

---

## 4. Priority Tiers

### Tier 1: Always Run (Every Validation)

Every validation regardless of depth setting.

**Rationale:** These catch bugs that will cause runtime failures or are security-adjacent. High signal, low noise.

| Check | Tool(s) | Category | CWE |
|-------|---------|----------|-----|
| Undefined names / compile errors | Ruff `F821`, `E999`, `E901` | CAT-01 | CWE-561, CWE-665 |
| Unused imports | Ruff `F401` | CAT-09 | CWE-561 |
| Mutable default arguments | Ruff `B006`, `RUF009`, `RUF012` | CAT-01, CAT-08 | CWE-665 |
| Blind exception swallowing | Ruff `BLE001`, `S110`; Pylint `W0702` | CAT-02 | CWE-390, CWE-703 |
| Empty catch block | PSScriptAnalyzer `PSAvoidUsingEmptyCatchBlock` | CAT-02 | CWE-390 |
| HTTP requests without timeout | Ruff `S113`; Pylint `W3101` | CAT-03 | CWE-1088 |
| Blocking call in async function | Ruff `ASYNC210`, `ASYNC251` | CAT-08 | CWE-1088 |
| Lost async task reference | Ruff `RUF006` | CAT-08 | CWE-404 |
| Assignment to automatic variables | PSScriptAnalyzer [REDACTED_SECRET] | CAT-01 | — |
| Invoke-Expression usage | PSScriptAnalyzer `PSAvoidUsingInvokeExpression` | CAT-01, Security | CWE-78, CWE-94 |
| Variable used before assignment | Pylint `E0601` | CAT-01 | CWE-665 |
| Dict mutation during iteration | Pylint `W4701`, `W4702` | CAT-01 | CWE-662 |
| Loop variable captured by closure | Ruff `B023`; Pylint `W0640` | CAT-01 | CWE-665 |
| **LLM: Error handling completeness** | LLM | CAT-02 | CWE-703 |
| **LLM: Logical correctness (obvious)** | LLM | CAT-01 | CWE-480, CWE-682 |

### Tier 2: Standard Depth

In addition to Tier 1. For standard and thorough validation.

| Check | Tool(s) | Category | CWE |
|-------|---------|----------|-----|
| Exception chaining | Ruff `B904`; Pylint `W0716` | CAT-02 | CWE-703 |
| Timezone-naive datetimes | Ruff `DTZ005`, `DTZ001` | CAT-07 | — |
| f-string in logging | Ruff `G004` | CAT-06 | CWE-1050 (perf) |
| Complexity metrics | Pylint `R0912`, `R0913`, `R0914`, `R0915` | CAT-04 | CWE-1064, CWE-1121 |
| Circular imports | Pylint `R0401` | CAT-09 | CWE-1047 |
| Naming conventions | Ruff `N801–N818`; Pylint `C0103` | CAT-06 | — |
| Missing docstrings | Ruff `D100–D107`; Pylint `C0114–C0116` | CAT-06 | — |
| Missing function help | PSScriptAnalyzer `PSProvideCommentHelp` | CAT-06 | — |
| Cmdlet aliases in scripts | PSScriptAnalyzer `PSAvoidUsingCmdletAliases` | CAT-06 | — |
| Positional parameters | PSScriptAnalyzer [REDACTED_SECRET] | CAT-06 | — |
| Async cancel scope errors | Ruff `ASYNC100` | CAT-08 | CWE-662 |
| Return value tracking | Ruff `RET501–508` | CAT-01 | CWE-252 |
| Type annotations presence | Ruff `ANN001–204` | CAT-07 | — |
| Print statements in non-scripts | Ruff `T20` | CAT-06 | — |
| **LLM: Resource lifecycle** | LLM | CAT-03 | CWE-404, CWE-772 |
| **LLM: Cohesion assessment** | LLM | CAT-04 | CWE-1083 |
| **LLM: Naming clarity** | LLM | CAT-06 | — |
| **LLM: Type annotation correctness** | LLM | CAT-07 | CWE-704 |
| **LLM: Testability assessment** | LLM | CAT-12 | — |

### Tier 3: Thorough Depth Only

Deep analysis. Only at `--depth thorough`. Higher cost, lower frequency.

| Check | Tool(s) | Category | CWE |
|-------|---------|----------|-----|
| Code duplication | Pylint `R0801` | CAT-05 | CWE-1041 |
| Excessive fan-out | Pylint `R0902` | CAT-04 | CWE-1048 |
| Portability analysis | Ruff `UP`, `YTT`; PSScriptAnalyzer `UseCompatible*` | CAT-11 | CWE-758 |
| Commented-out code | Ruff `ERA001` | CAT-06 | CWE-1085 |
| Unused arguments | Ruff `ARG001–005` | CAT-06 | CWE-1064 |
| **LLM: Near-duplicate logic** | LLM | CAT-05 | CWE-1041 |
| **LLM: Architecture/layer violations** | LLM | CAT-04 | CWE-1054 |
| **LLM: Hardcoded configuration review** | LLM | CAT-11 | CWE-1051, CWE-1052 |
| **LLM: PowerShell concurrency patterns** | LLM | CAT-08 | CWE-662 |
| **LLM: Test quality assessment** | LLM | CAT-12 | — |
| **LLM: Overall design coherence** | LLM | CAT-04 | — |

---

## 5. Agentic Code Patterns

Dedicated section for code patterns specific to AI/agentic systems. These patterns appear frequently in LLM-generated code, agent orchestration frameworks (LangChain, AutoGen, CrewAI), and AI infrastructure code.

---

### AP-01: Prompt Construction

**Risk:** Prompt injection, unintended behavior, costly API calls from malformed prompts.

#### Deterministic Checks

| Tool | Rule | What It Catches |
|------|------|----------------|
| Ruff | `S608` | SQL-injection pattern — same string interpolation anti-pattern applies to prompt construction |
| Ruff | `ISC001`, `ISC002` | Implicit string concatenation in prompt templates — accidental joins |
| Ruff | `B021` | f-string used as docstring — f-strings in unexpected string positions |

#### LLM Judgment Checks

| Check | What to Look For |
|-------|-----------------|
| Unsanitized user input in prompts | User-provided text directly interpolated into `system` or `user` messages without sanitization. Look for `f"...{user_input}..."` patterns flowing to LLM API calls. |
| Role boundary enforcement | Does the prompt enforce clear separation between system instructions and user content? Concatenated system + user messages are prompt injection vectors. |
| Prompt template exfiltration risk | Does the system prompt include instructions the user should not see? Is it protected from "repeat your instructions" attacks? |
| Jinja2 / template engine use | Any `Environment(autoescape=False)` with user-influenced template variables — XSS and injection risk (Ruff `S701`). |
| Token budget management | Prompts constructed without length checks — can exceed model context limits causing silent truncation. |
| Structured output schemas | Does the code validate that LLM responses conform to expected JSON schema before processing? |

---

### AP-02: LLM API Call Patterns

**Risk:** Hung agents, cost overruns, silent failures, data leakage.

#### Deterministic Checks

| Tool | Rule | What It Catches |
|------|------|----------------|
| Ruff | `S113` | HTTP request without timeout — all LLM API calls are HTTP |
| Ruff | `ASYNC210` | Synchronous HTTP call in async agent — blocks entire event loop |
| Ruff | `S501`, `S323` | SSL verification disabled — MITM risk on API traffic |
| Ruff | `S105–S107` | Hardcoded API keys (OpenAI, Anthropic, etc.) |
| Pylint | `W3101` | `requests` calls without timeout parameter |

#### LLM Judgment Checks

| Check | What to Look For |
|-------|-----------------|
| Retry logic completeness | Are transient API errors (429 rate limit, 503 service unavailable) retried with exponential backoff? Or do they propagate immediately as unhandled exceptions? |
| Model version pinning | Is `model="gpt-4"` or `model="gpt-4-turbo-2024-04-09"` used? Unpinned model names are subject to silent capability changes. |
| Response validation | Is the LLM response checked for expected structure before accessing fields? `response.choices[0].message.content` fails if API returns an error structure. |
| Cost controls | Are `max_tokens` set to prevent runaway generation? Are token counts tracked? |
| Streaming response handling | If streaming, are incomplete responses handled? Is the stream closed properly on error? |
| API key scope | Is the API key scoped to minimum permissions? Is it read from environment variable or secrets manager, not source? |

---

### AP-03: Error Handling in Agent Pipelines

**Risk:** Silent failures that appear as success; cascading failures that bring down entire pipelines.

#### Deterministic Checks

| Tool | Rule | What It Catches |
|------|------|----------------|
| Ruff | `BLE001` | Blind `except:` in agent tool handlers |
| Ruff | `S110` | `try-except-pass` in agent pipeline stages |
| Ruff | `B904` | Exception chaining loss in agent orchestration |
| Ruff | `TRY400` | `logging.error()` instead of `logging.exception()` — loses traceback |
| Pylint | `W0718` `broad-exception-caught` | `except Exception:` hiding tool call failures |

#### LLM Judgment Checks

| Check | What to Look For |
|-------|-----------------|
| Partial success handling | Does the pipeline handle the case where some tools succeed and others fail? Is partial state rolled back or tracked? |
| Structured error returns vs. exceptions | Agent frameworks often expect tools to return error dicts rather than raise exceptions. Is the contract consistent? |
| Timeout vs. exception consistency | When a tool times out, is it treated the same as a failed tool call, or does it propagate differently? |
| Fallback behavior | Does the agent have a meaningful fallback when a tool fails? Or does it silently proceed with missing data? |
| Error context preservation | When errors are logged, do they include the input that caused the failure, the tool that failed, and the pipeline stage? |
| `asyncio.CancelledError` propagation | This must NOT be swallowed. Look for `except Exception:` blocks that would catch it in async agent code. |

---

### AP-04: Credential & Secret Management

**Risk:** Credential exposure in code, logs, process lists, version control.  
**Delegation:** This section identifies credential management patterns for pre-screening. Security assessment of findings is owned by SEC-07 in `SECURITY_CRITIC_FRAMEWORK.md`. Code Hygiene flags the pattern; the Security Critic assesses severity, exploitability, and remediation.

#### Deterministic Checks

| Tool | Rule | What It Catches |
|------|------|----------------|
| Ruff | `S105–S107` | Hardcoded password/secret patterns |
| Ruff | `S108` | Hardcoded temp file paths (secrets written to predictable locations) |
| PSScriptAnalyzer | `PSAvoidUsingConvertToSecureStringWithPlainText` | Plaintext secrets in PowerShell automation |
| PSScriptAnalyzer | `PSAvoidUsingUsernameAndPasswordParams` | String-typed credential parameters |
| PSScriptAnalyzer | `PSAvoidUsingPlainTextForPassword` | Parameter names signaling plaintext credentials |
| PSScriptAnalyzer | `PSUsePSCredentialType` | Credential parameters should be `[PSCredential]` type |

#### LLM Judgment Checks

| Check | What to Look For |
|-------|-----------------|
| Environment variable usage | Are secrets loaded from `os.environ.get()` with no fallback defaults that expose the secret? (e.g., `os.environ.get("API_KEY", "sk-hardcoded-fallback")`) |
| Secrets in log output | Are credentials, tokens, or API keys logged at any log level, including DEBUG? |
| Secrets in exception messages | Exception messages that include credential values (`f"Failed to authenticate with key {api_key}"`) |
| Secrets in function signatures | API keys passed as function parameters vs. injected via configuration objects |
| Version control risk | Are `.env` files or config files with secrets excluded from version control (`.gitignore`)? |
| PowerShell SecureString adequacy | Is `SecureString` used only as a type wrapper while the value is immediately `ConvertFrom-SecureString` back to plaintext? |

---

### AP-05: Timeout & Retry Logic

**Risk:** Hung agents, no forward progress, resource exhaustion from infinite retries.

#### Deterministic Checks

| Tool | Rule | What It Catches |
|------|------|----------------|
| Ruff | `S113` | HTTP requests without timeout |
| Ruff | `ASYNC100` | Timeout scope with no awaitable inside — timeout never fires |
| Ruff | `ASYNC110` | Busy-wait polling loop — use `asyncio.Event` |
| Pylint | `W3101` | `requests` without timeout |

#### LLM Judgment Checks

| Check | What to Look For |
|-------|-----------------|
| Retry with exponential backoff | Is backoff calculated correctly? Is there a maximum retry count? Is there jitter to avoid thundering herd? |
| Retry on correct exception types | Retrying on `ValueError` (logic error) instead of `ConnectionError`/`TimeoutError` wastes calls and can loop forever. |
| Circuit breaker pattern | For high-volume agent systems, is there a circuit breaker that stops retrying a consistently failing service? |
| Timeout scope completeness | Does the timeout cover the entire operation (including connection + read), or just one phase? |
| Retry state isolation | Does retry logic correctly reset or reuse state between attempts? |
| PowerShell retry patterns | `Invoke-WebRequest` / `Invoke-RestMethod` — `-TimeoutSec` present? Retry loop uses `$ErrorActionPreference = 'Stop'`? |

---

### AP-06: Logging & Observability

**Risk:** Undebuggable agents; production failures that leave no trace.

#### Deterministic Checks

| Tool | Rule | What It Catches |
|------|------|----------------|
| Ruff | `LOG001` | Direct `Logger()` instantiation instead of `getLogger()` |
| Ruff | `LOG002` | `getLogger()` not using `__name__` — breaks logger hierarchy |
| Ruff | `LOG007` | `.exception()` with `exc_info=False` — loses traceback |
| Ruff | `LOG015` | Logging on root logger — hard to filter |
| Ruff | `G004` | f-string in log call — eagerly evaluated even when level is disabled |
| Pylint | `W1201` `logging-not-lazy` | `%` format evaluated eagerly in log calls |
| PSScriptAnalyzer | `PSAvoidUsingWriteHost` | `Write-Host` bypasses pipeline; use `Write-Verbose` / `Write-Information` |

#### LLM Judgment Checks

| Check | What to Look For |
|-------|-----------------|
| Structured logging for agent decisions | Does the agent log which tool it chose and why? Which prompt template was used? What the LLM returned before parsing? |
| Log level appropriateness | Is DEBUG used for verbose agent state, INFO for significant events, ERROR for failures — or is everything at one level? |
| Correlation IDs | For multi-step agent pipelines, is there a request/run ID that threads through all log entries? |
| Sensitive data in logs | PII, credentials, or full prompt content in log output — even at DEBUG level. |
| Log coverage at failure points | Are all exception handlers logging before re-raising or returning error? |
| PowerShell transcript risk | Does the script log sensitive data that would appear in PowerShell transcripts if enabled? |

---

## 6. Framework Citation Index

Quick reference: evaluation category → source frameworks.

| Category | ISO 25010 Sub-characteristics | CISQ CWEs | 
|----------|-------------------------------|-----------|
| CAT-01: Code Correctness | Reliability → Faultlessness | CWE-252, CWE-390, CWE-480, CWE-561, CWE-570, CWE-571, CWE-703 |
| CAT-02: Error Handling | Reliability → Fault Tolerance | CWE-248, CWE-252, CWE-390, CWE-391, CWE-703 |
| CAT-03: Resource Management | Performance Efficiency → Resource Utilization | CWE-401, CWE-404, CWE-459, CWE-672, CWE-772, CWE-1088 |
| CAT-04: Complexity & Modularity | Maintainability → Analysability, Modularity, Modifiability | CWE-407, CWE-1047, CWE-1048, CWE-1054, CWE-1064, CWE-1080, CWE-1121 |
| CAT-05: Code Duplication | Maintainability → Reusability | CWE-1041 |
| CAT-06: Naming & Documentation | Maintainability → Analysability; Interaction Capability → Self-Descriptiveness | CWE-1052, CWE-1085 |
| CAT-07: Type Safety | Reliability → Faultlessness | CWE-681, CWE-704, CWE-758 |
| CAT-08: Async & Concurrency | Reliability → Fault Tolerance, Faultlessness | CWE-366, CWE-662, CWE-833, CWE-835, CWE-1088 |
| CAT-09: Import Hygiene | Maintainability → Modularity | CWE-1047, CWE-1048 |
| CAT-10: Style & Formatting | Maintainability → Analysability | CWE-1080, CWE-1085 |
| CAT-11: Portability | Flexibility → Adaptability; Compatibility → Coexistence | CWE-758, CWE-1051 |
| CAT-12: Testability | Maintainability → Testability | (Inferred from CWE-1047 coupling) |
| AP-01: Prompt Construction | Reliability → Fault Tolerance; Security → Integrity | CWE-20, CWE-94 |
| AP-02: LLM API Calls | Reliability → Availability, Fault Tolerance | CWE-252, CWE-1088 |
| AP-03: Agent Pipeline Errors | Reliability → Fault Tolerance, Recoverability | CWE-248, CWE-390, CWE-703 |
| AP-04: Credential Management | Security → Confidentiality | CWE-259, CWE-321, CWE-798 |
| AP-05: Timeout & Retry | Reliability → Availability | CWE-1088, CWE-835 |
| AP-06: Logging & Observability | Security → Non-repudiation; Maintainability → Analysability | CWE-778 |

---

### SEI CERT Applicability Note

SEI CERT Coding Standards cover C, C++, Java, Perl, and Android — **there is no official CERT standard for Python or PowerShell**. Where CERT principles transfer by analogy (e.g., FIO02-C for path canonicalization, ENV33-C for command injection, MSC30-C for random number generation), this framework cites them as `CERT XXX-C analog` to indicate the principle applies but no official rule exists for the target language. CERT is not cited as a grounding standard for Python/PowerShell quality assessment. See `SECURITY_CRITIC_FRAMEWORK.md` §6 for the full CERT applicability analysis.

---

## 7. Delegation Severity Cap

**Design intent:** Code Hygiene assigns **LOW or MEDIUM severity only** for security-adjacent
patterns (eval/exec, Invoke-Expression, hardcoded credentials, unsanitized prompt inputs).
This is intentional — security severity, exploitability, and remediation assessment for these
patterns is delegated to the Security Critic.

**Rationale:** Assigning HIGH/CRITICAL from the Code Hygiene Critic for patterns that the
Security Critic also evaluates would create artificial severity inflation through aggregation.
The Code Hygiene finding exists for traceability (code is hard to audit/test); the Security
Critic provides the risk verdict.

**Finding format for delegated patterns:**
```
"Code hygiene concern: [pattern] is difficult to audit and test.
Security severity and exploitability assessment delegated to SecurityCritic."
Severity: LOW or MEDIUM (hygiene concern only)
```

---

*Framework version: 1.1 · Updated: 2026-03-06*  
*Sources: ISO/IEC 25010:2023, ISO/IEC 5055:2021 (CISQ/OMG ASCQM), Ruff v0.9.x rule set, Pylint v4.0.x checker docs, PSScriptAnalyzer latest*


path: docs/critics/POWERSHELL_COVERAGE.md
content: # PowerShell Coverage — Honest Assessment

**Status:** v0.5.2  
**Last Updated:** March 2026

---

## Summary

Quorum's Security and Code Hygiene critics evaluate both Python and PowerShell code. **Python coverage is stronger (~85%+) due to a richer SAST ecosystem. PowerShell coverage is approximately 70%**, with known gaps documented below.

This document exists because we believe honest disclosure of coverage limitations is more valuable than pretending they don't exist. If you're using Quorum to evaluate PowerShell code, you should know exactly where the blind spots are.

---

## What Works Well

### PSScriptAnalyzer Integration (Deterministic Pre-Screen)

The following PSSA rules are integrated or planned for the pre-screen layer:

| Category | Rules | Status |
|----------|-------|--------|
| **Credential handling** | `PSAvoidUsingConvertToSecureStringWithPlainText`, `PSAvoidUsingUsernameAndPasswordParams`, `PSAvoidUsingPlainTextForPassword`, `PSUsePSCredentialType` | ✅ LLM-evaluated |
| **Injection** | `PSAvoidUsingInvokeExpression` | ✅ LLM-evaluated |
| **Crypto** | `PSAvoidUsingBrokenHashAlgorithms` | ✅ LLM-evaluated |
| **Error handling** | `PSAvoidUsingEmptyCatchBlock` | ✅ LLM-evaluated |
| **Network security** | `PSAvoidUsingAllowUnencryptedAuthentication` | ✅ LLM-evaluated |
| **Infrastructure** | `PSAvoidUsingComputerNameHardcoded` | ✅ LLM-evaluated |

These rules provide solid coverage for the most common PowerShell security anti-patterns.

### LLM Semantic Analysis

The Security Critic's LLM judgment checks work equally well on PowerShell and Python for:

- Authorization logic review (SEC-04)
- Business logic validation (SEC-02)  
- Certificate validation bypass patterns
- Download cradle detection (`iwr | iex`, `DownloadString`)
- AMSI bypass pattern recognition
- Registry persistence detection
- `[ScriptBlock]::Create()` with user-controlled input
- Path traversal in `Get-Content`/`Set-Content`

---

## Known Gaps

### No PSSA Rules Exist For:

| Gap | Security Impact | Workaround |
|-----|----------------|------------|
| **SQL injection via `Invoke-Sqlcmd`** | T1 — High | LLM detection only; no deterministic pre-screen |
| **Path traversal** | T2 — Medium | LLM detection only |
| **SSRF via `Invoke-WebRequest`** | T2 — Medium | LLM detection only |
| **Deserialization (`Import-CliXml`, `ConvertFrom-Json` type hydration)** | T3 — Low-Medium | LLM detection only |
| **Session management** | T2 — Medium | LLM detection only |
| **DoS / resource consumption** | T3 — Low | LLM detection only |
| **Missing `-ErrorAction Stop` in try/catch** | T2 — Medium | LLM detection only |
| **`-TimeoutSec` on web requests** | T1 — Medium | LLM detection only |

### Ecosystem Gap: Why PowerShell Lags

PSScriptAnalyzer is the only mainstream PowerShell SAST tool. It has ~60 rules focused primarily on style and best practices, with limited security coverage. By comparison, Python has:

- **Ruff:** 80+ security rules (Bandit-derived `S*` series)
- **Bandit:** Dedicated Python security linter
- **Semgrep/CodeQL:** Deep taint analysis with Python support

PowerShell has no equivalent to Bandit, and PSScriptAnalyzer's security rules are mostly credential-focused. This is a tooling ecosystem gap, not a Quorum limitation — but it means PowerShell security evaluation leans more heavily on LLM judgment than deterministic checks.

### What This Means in Practice

For **Python** code: ~60% of findings come from deterministic pre-screen (SAST), ~40% from LLM judgment.  
For **PowerShell** code: ~25% of findings come from deterministic checks (PSSA), ~75% from LLM judgment.

LLM-only findings are still grounded in framework citations (OWASP ASVS, CWE, SA-11) and require evidence. They are not less valid — but they lack the reproducibility guarantee of deterministic checks. Two runs of the same PowerShell file may surface different LLM findings, while SAST findings are identical every time.

---

## Improvement Roadmap

1. **Microsoft DevSkim integration** — MIT-licensed, CLI-invokable (`devskim analyze`), SARIF/JSON output. Adds ~15 cross-language security patterns (dangerous functions, hardcoded URLs, credential patterns). Complementary to PSSA, low integration effort.
2. **Custom PSScriptAnalyzer rules** — Write ~10-15 custom PSSA rules for the highest-impact gaps (SQL injection via `Invoke-Sqlcmd`, path traversal in `Get-Content`, `Start-Process` with unsanitized input). PSSA supports custom rules in PowerShell or compiled C# — this is the highest-leverage improvement.
3. **Pre-screen expansion** — Add regex-based deterministic checks for PowerShell patterns that don't need full PSSA (e.g., `Invoke-Sqlcmd` + string concatenation)
4. **Community contribution** — Contribute security rules upstream to PSScriptAnalyzer

**Not on the horizon:** Semgrep and CodeQL have both confirmed PowerShell is not on their roadmaps (as of early 2026). No maintained community security rule packs for PSSA exist. The gap is structural — no major SAST engine has built a PowerShell AST parser outside of PSSA's .NET implementation.

---

## Recommendation

If you're evaluating **security-critical PowerShell** code with Quorum:

1. Run at **standard** or **thorough** depth — quick depth may miss LLM-only findings
2. Cross-reference findings with a manual review for SEC-04 (authorization) and SEC-01 (injection) — the two categories with the highest LLM-only dependency
3. Consider supplementing with ScriptAnalyzer custom rules for your specific environment
4. Treat the PowerShell assessment as a strong starting point, not a complete audit

---

> ⚖️ **LICENSE** — This file is part of [Quorum](https://github.com/SharedIntellect/quorum).  
> Copyright 2026 SharedIntellect. MIT License.  
> See [LICENSE](https://github.com/SharedIntellect/quorum/blob/main/LICENSE) for full terms.


path: docs/critics/SEC02_BUSINESS_LOGIC_VALIDATION.md
content: # SEC-02: Business Logic Validation Workflow

**Status:** 🔜 Documented workflow — runtime integration planned  
**Date:** March 2026  
**Framework Basis:** OWASP ASVS 5.0.0 V2 (Validation and Business Logic), CWE-20, CWE-606

---

## The Problem

SEC-02 (Input Validation & Sanitization) covers two distinct concerns:

1. **Technical validation** — type checking, format enforcement, bounds checking. SAST and regex patterns handle this reasonably well.
2. **Business logic validation** — does the code enforce domain-specific rules? Can a user order -1 items? Set a future birth date? Access another user's data through a valid but unauthorized input?

SAST tools are blind to business logic. They can't know that a negative quantity is invalid unless someone encodes that rule. This is the Security Critic's highest-value contribution in SEC-02 — and the area where it needs the most guidance to be useful.

---

## Requirements → Critic Path

The workflow for business logic validation follows a three-stage pipeline:

### Stage 1: Requirements Capture (User-Provided)

Business rules must come from somewhere. Quorum does not invent them — they are declared by the user through one of three mechanisms:

**Option A: Rubric Criteria (Recommended)**

Encode business rules directly as rubric criteria:

```json
{
  "id": "BL-001",
  "criterion": "Order quantities must be positive integers between 1 and 9999",
  "severity": "HIGH",
  "evidence_type": "tool",
  "evidence_instruction": "Search for order quantity handling. Verify bounds checking exists before the value reaches business logic or database.",
  "framework_refs": ["CWE-20", "ASVS V2.2.*"]
}
```

This is the most explicit path. Each business rule becomes a testable criterion with clear evidence requirements.

**Option B: Relationship Manifest — Threat Context**

Use the `threat_context` relationship type to declare sensitive operations and roles:

```yaml
- type: threat_context
  target: app/orders/views.py
  context:
    sensitive_operations:
      - "Order creation (quantity must be positive, price must match catalog)"
      - "Discount application (percentage must be 0-100, requires manager role)"
      - "Refund processing (amount cannot exceed original order total)"
```

The Security Critic receives this context and evaluates whether the code enforces these constraints. Less precise than rubric criteria, but lower friction for initial assessment.

**Option C: Inline Documentation**

If the codebase uses docstrings, comments, or type annotations to express business rules, the Security Critic can extract and verify them:

```python
def process_order(quantity: int, price: Decimal) -> Order:
    """Create a new order.
    
    Business rules:
    - quantity must be 1-9999
    - price must match catalog price (no client-side price override)
    - total must not exceed account credit limit
    """
```

This is the weakest path — it depends on documentation accuracy and completeness. But it's better than nothing when rubrics and manifests aren't available.

**When Option C works well:** Docstrings follow a consistent pattern (e.g., `Business rules:` header), rules are co-located with the code they govern, and the codebase has reasonable documentation coverage.

**When Option C produces junk findings:** Business rules are scattered across comments, commit messages, Slack threads, or tribal knowledge. The critic finds docstrings but they describe *what* the function does, not *what constraints it must enforce*. If your inline docs don't express rules, use Option A or B instead.

### Composing Relationship Types — Full Manifest Example

A single `quorum-relationships.yaml` can combine `implements`, `threat_context`, and other relationship types to give critics a complete picture:

```yaml
# quorum-relationships.yaml — e-commerce order service
relationships:
  # Code implements the order specification
  - type: implements
    source: docs/order-spec.md
    target: app/orders/views.py

  # Threat model context for authorization + business logic
  - type: threat_context
    target: app/orders/views.py
    context:
      sensitive_operations:
        - "Order creation (quantity 1-9999, price from catalog only)"
        - "Discount application (0-100%, manager role required)"
        - "Refund processing (cannot exceed original total)"
      roles: ["customer", "manager", "admin"]
      trust_boundary: "All inputs cross trust boundary from public API"

  # Schema contract between API and database
  - type: schema_contract
    source: app/orders/serializers.py
    target: app/orders/models.py

  # Documents relationship for cross-artifact consistency
  - type: documents
    source: docs/order-spec.md
    target: app/orders/models.py
```

This manifest feeds the Security Critic (SEC-02 business logic + SEC-04 authorization), the Correctness Critic (implements relationship), and the Cross-Artifact Consistency check (all relationships). One file, multiple critics, composable.

### Stage 2: Critic Evaluation

The Security Critic evaluates business logic validation using this checklist:

| Check | What the Critic Looks For | Evidence Required |
|-------|--------------------------|-------------------|
| **Boundary enforcement** | Are numeric inputs bounds-checked before use? | Code showing validation + the boundary values |
| **State transition validity** | Can entities move to invalid states? (e.g., cancelled order → shipped) | State machine or status field handling code |
| **Cross-field consistency** | Are related fields validated together? (e.g., start_date < end_date) | Validation logic that checks field relationships |
| **Negative/zero handling** | Do quantities, amounts, and counts reject non-positive values? | Input validation before business logic |
| **Privilege-gated operations** | Are sensitive operations (refund, delete, admin actions) role-checked? | Authorization check before the operation |
| **Idempotency** | Can the same request be safely repeated? Double-submit protection? | Idempotency keys or duplicate detection |
| **Rate limiting on business operations** | Can a user trigger expensive operations without throttling? | Rate limit or queue mechanism |

### Severity Calibration

Not all business logic gaps carry equal weight. Use these defaults unless domain context overrides:

| Check | Typical Severity | Rationale |
|-------|-----------------|-----------|
| **Boundary enforcement** | **HIGH** | Direct data corruption, financial impact (negative quantities, overflow) |
| **State transition validity** | **HIGH** | Invalid state transitions can be irreversible and break downstream workflows |
| **Cross-field consistency** | **MEDIUM** | Usually caught by downstream validation or database constraints; becomes HIGH if financial |
| **Negative/zero handling** | **HIGH** when financial, **MEDIUM** otherwise | A negative dollar amount is worse than a negative page count |
| **Privilege-gated operations** | **CRITICAL** if admin/financial, **HIGH** otherwise | Missing authorization on sensitive ops is always severe |
| **Idempotency** | **MEDIUM** | Duplicated operations are usually recoverable; becomes **HIGH** if financial (double-charge) |
| **Rate limiting** | **MEDIUM** | Abuse potential varies; becomes **HIGH** if expensive operations (bulk email, payment processing) |

The critic uses these defaults when no explicit severity is declared in rubric criteria. Override by setting `"severity"` in your rubric criterion definition.

### Stage 3: Finding Generation

Business logic findings use the standard finding format with SEC-02 category:

```json
{
  "issue_id": "SEC-02-BL-001",
  "critic": "security",
  "severity": "HIGH",
  "category": "SEC-02",
  "description": "Order quantity accepts negative values — no bounds check before database write",
  "evidence": {
    "type": "grep",
    "output": "def create_order(qty): Order.objects.create(quantity=qty)  # line 47",
    "tool_command": "grep -n 'create_order\\|quantity' app/orders/views.py"
  },
  "framework_refs": ["CWE-20", "ASVS V2.2.*", "CWE-606"],
  "remediation": "Add validation: if qty < 1 or qty > 9999: raise ValidationError"
}
```

---

## What the Critic Cannot Do (Honest Limitations)

1. **Invent business rules.** Without requirements (rubric, manifest, or docs), the critic can only flag obvious anti-patterns (negative quantities, missing null checks). It cannot determine that "maximum order value is $50,000" unless told.

2. **Validate complex state machines.** Multi-step workflows with branching transitions require more context than a single code review provides. The critic can spot missing guards but cannot verify the full state graph.

3. **Assess domain correctness.** The critic doesn't know if a 15% discount is reasonable or a data entry error. Domain-specific reasonableness checks require human judgment or domain-specific rubrics.

4. **Replace integration testing.** Business logic validation in code review catches structural gaps. It does not replace testing that exercises the actual execution paths.

---

## Integration with Other Categories

Business logic validation intersects with:

| Category | Intersection |
|----------|-------------|
| **SEC-04 (Authorization)** | Privilege-gated operations are both authorization and business logic. The `threat_context` manifest serves both. |
| **SEC-01 (Injection)** | Input that passes business validation but contains injection payloads. Business validation runs first; sanitization must also run. |
| **SEC-08 (Path Traversal)** | File paths that are valid business inputs but traverse outside allowed directories. |
| **SEC-14 (DoS)** | Business operations that are individually valid but can be abused at volume (e.g., bulk order creation without rate limiting). |

---

## Depth Levels and SEC-02

Business logic checks are available at **standard** and **thorough** depth:

| Depth | SEC-02 Coverage | Notes |
|-------|----------------|-------|
| **quick** | ❌ Not run | Quick runs only pre-screen + Correctness + Completeness. Security critic is not dispatched. |
| **standard** | ✅ Full SEC-02 | Security critic runs with all business logic checks. Rubric criteria and threat_context are evaluated. No fix loops. |
| **thorough** | ✅ Full SEC-02 + fix loop | Same coverage as standard, plus the Fixer proposes remediations for CRITICAL/HIGH findings and re-validates. |

If you need business logic validation, use `--depth standard` at minimum. The `quick` depth is designed for fast feedback on obvious issues, not comprehensive security review.

---

## Recommended Approach for Assessments

1. **Start with rubric criteria** — if you know the business rules, encode them. This gives the best results.
2. **Add threat context** — declare sensitive operations in the relationship manifest. This gives the critic enough to evaluate authorization and basic validation.
3. **Review findings critically** — business logic findings from LLM analysis are inherently more speculative than SAST findings. Cross-reference with domain experts.
4. **Iterate** — the first assessment reveals gaps in your requirements capture. Each subsequent run gets tighter.

---

> ⚖️ **LICENSE** — This file is part of [Quorum](https://github.com/SharedIntellect/quorum).  
> Copyright 2026 SharedIntellect. MIT License.  
> See [LICENSE](https://github.com/SharedIntellect/quorum/blob/main/LICENSE) for full terms.


path: docs/critics/SEC05_PIPELINE_RESILIENCE.md
content: # SEC-05: Pipeline Resilience and Adversarial Inputs

**Status:** Stub
**Author:** Akkari (with Daniel Cervera)
**Date:** 2026-03-09
**Scope:** Threat model for attacks targeting the validation pipeline's operational integrity — distinct from evidence fabrication (SEC-03)

**Origin:** Identified during SEC-03 review. The question "has there ever been a buffer overflow attack via feeding an AI model?" surfaced a class of threats where the goal isn't to pass validation but to *break or degrade* it.

---

## Distinction from SEC-03

**SEC-03** asks: *Is the evidence real, adequate, and authoritative?*
**SEC-05** asks: *Can the pipeline itself be subverted, exhausted, or degraded?*

SEC-03 threats want a PASS. SEC-05 threats may want a failure — denial of validation is itself a win if it blocks or delays an assessment.

---

## Attack Vectors (to develop)

### Context Window Exhaustion
Artifact so large it fills the critic's context window, pushing rubric criteria and system prompt out of effective attention. Analogous to buffer overflow in spirit — overwhelm a fixed-capacity resource to bypass controls.

### Parser Exploitation
Deeply nested or malformed YAML/JSON/structured data designed to crash or hang the pre-screen layer before critics are ever invoked.

### Output Corruption
Artifact content designed to make critics produce malformed output that the aggregator can't parse — causing crashes, timeouts, or silent null verdicts propagating downstream.

### Cost/Budget Exhaustion
Repeated or oversized submissions designed to burn through `--max-cost` budget on garbage artifacts, leaving no budget for real validation. A resource exhaustion attack on the economic layer.

### Denial of Validation
Any combination of the above used not to pass, but to prevent or delay a legitimate assessment from completing.

---

## Existing Defenses (audit needed)

- `--max-cost` cap (limits economic exposure)
- Crash-resilient batch with `--resume` and SIGTERM handler (limits blast radius)
- Pre-screen runs before LLM critics (deterministic gate)
- Timeout on individual critic runs

## Open Questions

- What are the actual context window limits per model, and does the pipeline enforce artifact size limits before sending to critics?
- Can a malformed artifact cause the aggregator to produce a verdict of None? (Possible connection to the verdict.json None bug)
- Should the pre-screen include an artifact size/complexity gate?
- How does `--max-cost` interact with adversarial inputs that are cheap to submit but expensive to evaluate?

---

## References

- SEC-03 — Evidence Integrity and Anti-Gaming (companion threat model)
- SEC-04 — Threat context relationships
- Anthropic, "Many-shot Jailbreaking" (2024) — context window manipulation research
- OWASP LLM Top 10 — LLM01 (Prompt Injection), LLM04 (Denial of Service)


path: docs/critics/SEC07_SELF_VALIDATION_CONVERGENCE.md
content: # SEC-07: Self-Validation Convergence Problem

**Status:** Observed, documenting
**Author:** Akkari (with Daniel Cervera)
**Date:** 2026-03-10
**Scope:** Known pattern where iterative self-validation fails to converge to zero findings

**Origin:** Observed during PR #8 (Tester critic). Three rounds of GRAD self-validation fixes, each resolving findings but generating new ones. Tests passed consistently; self-validation never stabilized.

---

## 1. Problem Statement

When Quorum validates its own code via CI (the GRAD workflow), fixing findings from one run can generate new findings in the next. The cycle does not reliably converge to zero findings, creating a potential infinite loop that blocks merging even when all tests pass.

This is a product-level concern — any user who enables Quorum self-validation as a required CI check will encounter this pattern.

## 2. Why It Happens

### 2.1 Fresh Context Per Run

Each CI run evaluates the code with no memory of prior runs. A fix that resolved a HIGH finding may introduce code patterns that trigger new MEDIUM findings. The critic doesn't know the previous code was worse — it only sees the current state.

### 2.2 Fix-Induced Findings

Defensive code patterns recommended by the critic can themselves be flagged:
- Narrowing `except Exception` to specific types → critic flags "does this cover all failure modes?"
- Adding explicit `None` checks → critic flags "redundant guard" or "information disclosure in error path"
- Sanitizing error messages → critic flags "lost exception context"

The recommended fix for finding A becomes the trigger for finding B.

### 2.3 Severity Recalibration

Fixing a CRITICAL or HIGH finding can cause the critic to reassess the surrounding code at a higher standard, surfacing MEDIUMs and LOWs that were previously shadowed by the more severe finding.

### 2.4 No Delta Awareness

The CI workflow evaluates all findings in changed files, not just *new* findings relative to the base branch. A file with 10 existing findings that gets a one-line fix will report all 10 + any new ones, making it impossible to distinguish progress from regression.

## 3. Impact

- **CI blocker:** If self-validation is a required status check, PRs with passing tests can be stuck indefinitely
- **Developer frustration:** Each "fix" round takes minutes of CI time and may produce more work than it resolves
- **False signal:** The pattern can erode trust in self-validation — developers learn to ignore or override it
- **Cost:** Each CI validation run consumes LLM tokens ($0.50–$2.00 at standard depth)

## 4. Candidate Mitigations (to explore)

### 4.1 Delta-Only Mode
Only report findings that are NEW relative to the base branch. Existing findings in changed files are suppressed. This is how most linters work (`--diff` mode) and prevents the "fix one, find ten" pattern.

**Effort:** Medium. Requires storing baseline findings per branch or computing a diff against the base branch's last validation run.

### 4.2 Convergence Detection
Track finding count across iterations. If the count isn't decreasing after N rounds, warn the user and offer a graceful override rather than a hard block.

**Effort:** Low. Requires a counter in the CI workflow and a conditional pass.

### 4.3 Severity-Based Gating
Only gate on CRITICAL findings. HIGH/MEDIUM/LOW are reported as advisory annotations on the PR but don't block merging.

**Effort:** Low. Workflow change only — check exit code against a severity threshold.

### 4.4 Finding Deduplication Across Runs
Hash each finding's core attributes (location, description pattern, criterion) and suppress re-flagged findings that were present in the previous run. Only surface genuinely new findings.

**Effort:** Medium-High. Requires persistent storage of finding hashes across CI runs (cache or artifact).

### 4.5 Max Iteration Cap
Allow N rounds of self-validation fixes. After N rounds, if findings remain, report them as known issues and allow merge with an annotation.

**Effort:** Low. Policy decision, not a code change.

## 5. Recommended Default

Until a proper delta mode is implemented:

- **Self-validation as advisory**, not required — report findings as PR comments but don't block merge
- **Tests as the hard gate** — all unit/integration tests must pass
- **CRITICAL-only blocking** — only CRITICAL self-validation findings block merge (§4.3)

This preserves the value of self-validation (visibility into code quality) without creating an unresolvable blocker.

## 6. References

- PR #8 (SharedIntellect/quorum) — first observed instance
- SEC-03 §3.3 — Assurance Level Model (gating thresholds should scale with assurance level)
- GRAD workflow (`.github/workflows/quorum-validate.yml`)


path: docs/getting-started/INSTALLATION.md
content: # Installation

## OpenClaw (Recommended)

### From ClawHub

```bash
openclaw skills add quorum
```

That's it. Quorum is now available to your agent. Ask it to run a quorum check on any file.

### From Source

Clone the repo and point OpenClaw at it:

```bash
git clone https://github.com/SharedIntellect/quorum.git
cd quorum
```

The reference implementation lives in `reference-implementation/`. You can run it directly:

```bash
cd reference-implementation
pip install -r requirements.txt
python -m quorum.cli run examples/sample-config.yaml --rubric examples/rubrics/research-quality.json
```

## Verify It Works

The quickest way to confirm everything is set up:

```bash
# From the reference-implementation directory:
python -m quorum.cli run examples/sample-config.yaml --rubric examples/rubrics/research-quality.json
```

You should see:
1. First-run config setup (if no `quorum-config.yaml` exists) — it'll ask your preferred models
2. Critics spawning and evaluating the sample artifact
3. A structured verdict with findings and evidence citations

If you see a `PASS`, `PASS_WITH_NOTES`, or `FAIL` verdict with cited evidence, Quorum is working.

## Configuration

After first run, Quorum creates `quorum-config.yaml` in your working directory. Edit it to change models, depth profiles, or critic selection:

```yaml
models:
  tier_1: anthropic/claude-sonnet-4-6
  tier_2: anthropic/claude-sonnet-4-6

depth: standard    # quick | standard | thorough
```

See [CONFIG_REFERENCE.md](../configuration/CONFIG_REFERENCE.md) for full options and [MODEL_REQUIREMENTS.md](MODEL_REQUIREMENTS.md) for supported models.

## Requirements

- **Python 3.10+**
- **An API key** for at least one supported model provider (Anthropic, OpenAI, or Google)
- **OpenClaw** (optional but recommended — handles agent orchestration automatically)
- **LiteLLM** (installed via `requirements.txt` — provides universal model access)

## Troubleshooting

**"No model configured"** — Run `quorum run` once and follow the first-run setup prompts, or manually create `quorum-config.yaml` (see [CONFIG_REFERENCE.md](../configuration/CONFIG_REFERENCE.md)).

**"API key not found"** — Set your provider's API key as an environment variable:
```bash
export ANTHROPIC_API_KEY=sk-ant-...
# or
export OPENAI_API_KEY=sk-...
```

**Critics timing out** — Large artifacts with `thorough` depth can take 45-90 minutes. Try `quick` depth first to verify setup, then scale up.

**"Model not supported"** — Check [MODEL_REQUIREMENTS.md](MODEL_REQUIREMENTS.md) for compatible models. Quorum requires models with strong reasoning and tool-use capabilities.

## Updating

### ClawHub
```bash
openclaw skills update quorum
```

### From Source
```bash
cd quorum
git pull origin main
pip install -r reference-implementation/requirements.txt
```


---

> ⚖️ **LICENSE** — Not part of the operational specification above.
> This file is part of [Quorum](https://github.com/SharedIntellect/quorum).
> Copyright 2026 SharedIntellect. MIT License.
> See [LICENSE](https://github.com/SharedIntellect/quorum/blob/main/LICENSE) for full terms.


path: reference-implementation/requirements-test.txt
content: pytest>=7.4.0
pytest-cov>=4.1.0
pytest-timeout>=2.1.0
coverage>=7.3.0
hypothesis>=6.92.0
freezegun>=1.5.0


Security scan verdicts — ClawScan: clean | static analysis: suspicious | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)