Skill: frmoretto/clarity-gate (version 2.1.3)

--- # agentskills.io compliant frontmatter name: clarity-gate version: 2.1.3 description: > Pre-ingestion verification for epistemic quality in RAG systems. Ensures documents are properly qualified before entering knowledge bases. Produces CGD (Clarity-Gated Documents) and validates SOT (Source of Truth) files. author: Francesco Marinoni Moretto license: CC-BY-4.0 repository: https://github.com/frmoretto/clarity-gate triggers: - clarity gate - check for hallucination risks - can an LLM read this safely - review for equivocation - verify document clarity - pre-ingestion check - cgd verify - sot verify capabilities: - document-verification - epistemic-quality - rag-preparation - cgd-generation - sot-validation outputs: - type: cgd extension: .cgd.md spec: docs/CLARITY_GATE_FORMAT_SPEC.md spec_version: "2.1" --- # Clarity Gate v2.1 **Purpose:** Pre-ingestion verification system that enforces epistemic quality before documents enter RAG knowledge bases. Produces Clarity-Gated Documents (CGD) compliant with the Clarity Gate Format Specification v2.1. **Core Question:** "If another LLM reads this document, will it mistake assumptions for facts?" **Core Principle:** *"Detection finds what is; enforcement ensures what should be. In practice: find the missing uncertainty markers before they become confident hallucinations."* --- ## What's New in v2.1 | Feature | Description | |---------|-------------| | **Claim Completion Status** | PENDING/VERIFIED determined by field presence (no explicit status field) | | **Source Field Semantics** | Actionable source (PENDING) vs. what-was-found (VERIFIED) | | **Claim ID Format Guidance** | Hash-based IDs preferred, collision analysis for scale | | **Body Structure Requirements** | HITL Verification Record section mandatory when claims exist | | **New Validation Codes** | E-ST10, W-ST11, W-HC01, W-HC02, E-SC06 (FORMAT_SPEC); E-TB01-07 (SOT validation) | | **Bundled Scripts** | `claim_id.py` and `document_hash.py` for deterministic computations | --- ## Specifications This skill implements and references: | Specification | Version | Location | |---------------|---------|----------| | Clarity Gate Format (Unified) | v2.1 | [docs/CLARITY_GATE_FORMAT_SPEC.md](../../docs/CLARITY_GATE_FORMAT_SPEC.md) | **Note:** v2.0 unifies CGD and SOT into a single `.cgd.md` format. SOT is now a CGD with an optional `tier:` block. --- ## Validation Codes Clarity Gate defines validation codes for structural and semantic checks per FORMAT_SPEC v2.1: ### HITL Claim Validation (§1.3.2-1.3.3) | Code | Check | Severity | |------|-------|----------| | **W-HC01** | Partial `confirmed-by`/`confirmed-date` fields | WARNING | | **W-HC02** | Vague source (e.g., "industry reports", "TBD") | WARNING | | **E-SC06** | Schema error in `hitl-claims` structure | ERROR | ### Body Structure (§1.2.1) | Code | Check | Severity | |------|-------|----------| | **E-ST10** | Missing `## HITL Verification Record` when claims exist | ERROR | | **W-ST11** | Table rows don't match `hitl-claims` count | WARNING | ### SOT Table Validation (§3.1) | Code | Check | Severity | |------|-------|----------| | **E-TB01** | No `## Verified Claims` section | ERROR | | **E-TB02** | Table has no data rows | ERROR | | **E-TB03** | Required columns missing | ERROR | | **E-TB04** | Column order wrong | ERROR | | **E-TB05** | Empty cell in required column | ERROR | | **E-TB06** | Invalid date format in Verified column | ERROR | | **E-TB07** | Verified date in future (beyond 24h grace) | ERROR | **Note:** Additional validation codes may be defined in RFC-001 (clarification document) but are not part of the normative FORMAT_SPEC. --- ## Bundled Scripts This skill includes Python scripts for deterministic computations per FORMAT_SPEC. ### scripts/claim_id.py Computes stable, hash-based claim IDs for HITL tracking (per §1.3.4). ```bash # Generate claim ID python scripts/claim_id.py "Base price is $99/mo" "api-pricing/1" # Output: claim-75fb137a # Run test vectors python scripts/claim_id.py --test ``` **Algorithm:** 1. Normalize text (strip + collapse whitespace) 2. Concatenate with location using pipe delimiter 3. SHA-256 hash, take first 8 hex chars 4. Prefix with "claim-" **Test vectors:** - `claim_id("Base price is $99/mo", "api-pricing/1")` → `claim-75fb137a` - `claim_id("The API supports GraphQL", "features/1")` → `claim-eb357742` ### scripts/document_hash.py Computes document SHA-256 hash per FORMAT_SPEC §2.2-2.4 with full canonicalization. ```bash # Compute hash python scripts/document_hash.py my-doc.cgd.md # Output: 7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730 # Verify existing hash python scripts/document_hash.py --verify my-doc.cgd.md # Output: PASS: Hash verified: 7d865e... # Run normalization tests python scripts/document_hash.py --test ``` **Algorithm (per §2.2-2.4):** 1. Extract content between opening `---\n` and `<!-- CLARITY_GATE_END -->` 2. Remove `document-sha256` line from YAML frontmatter ONLY (with multiline continuation support) 3. Canonicalize: - Strip trailing whitespace per line - Collapse 3+ consecutive newlines to 2 - Normalize final newline (exactly 1 LF) - UTF-8 NFC normalization 4. Compute SHA-256 **Cross-platform normalization:** - BOM removed if present - CRLF to LF (Windows) - CR to LF (old Mac) - Boundary detection (prevents hash computation on content outside CGD structure) - Whitespace variations produce identical hashes (deterministic across platforms) --- ## The Key Distinction Existing tools like UnScientify and HedgeHunter (CoNLL-2010) **detect** uncertainty markers already present in text ("Is uncertainty expressed?"). Clarity Gate **enforces** their presence where epistemically required ("Should uncertainty be expressed but isn't?"). | Tool Type | Question | Example | |-----------|----------|---------| | **Detection** | "Does this text contain hedges?" | UnScientify/HedgeHunter find "may", "possibly" | | **Enforcement** | "Should this claim be hedged but isn't?" | Clarity Gate flags "Revenue will be $50M" | --- ## Critical Limitation > **Clarity Gate verifies FORM, not TRUTH.** > > This skill checks whether claims are properly marked as uncertain—it cannot verify if claims are actually true. > > **Risk:** An LLM can hallucinate facts INTO a document, then "pass" Clarity Gate by adding source markers to false claims. > > **Solution:** HITL (Human-In-The-Loop) verification is **MANDATORY** before declaring PASS. --- ## When to Use - Before ingesting documents into RAG systems - Before sharing documents with other AI systems - After writing specifications, state docs, or methodology descriptions - When a document contains projections, estimates, or hypotheses - Before publishing claims that haven't been validated - When handing off documentation between LLM sessions --- ## The 9 Verification Points ### Relationship to Spec Suite The 9 Verification Points guide **semantic review** — content quality checks that require judgment (human or AI). They answer questions like "Should this claim be hedged?" and "Are these numbers consistent?" When review completes, output a CGD file conforming to [CLARITY_GATE_FORMAT_SPEC.md](../../docs/CLARITY_GATE_FORMAT_SPEC.md). The C/S rules in [CLARITY_GATE_FORMAT_SPEC.md](../../docs/CLARITY_GATE_FORMAT_SPEC.md) validate **file structure**, not semantic content. **The connection:** 1. Semantic findings (9 points) determine what issues exist 2. Issues are recorded in CGD state fields (`clarity-status`, `hitl-status`, `hitl-pending-count`) 3. State consistency is enforced by structural rules (C7-C10) *Example: If Point 5 (Data Consistency) finds conflicting numbers, you'd mark `clarity-status: UNCLEAR` until resolved. Rule C7 then ensures you can't claim `REVIEWED` while still `UNCLEAR`.* --- ### Epistemic Checks (Core Focus: Points 1-4) **1. HYPOTHESIS vs FACT LABELING** Every claim must be clearly marked as validated or hypothetical. | Fails | Passes | |-------|--------| | "Our architecture outperforms competitors" | "Our architecture outperforms competitors [benchmark data in Table 3]" | | "The model achieves 40% improvement" | "The model achieves 40% improvement [measured on dataset X]" | **Fix:** Add markers: "PROJECTED:", "HYPOTHESIS:", "UNTESTED:", "(estimated)", "~", "?" --- **2. UNCERTAINTY MARKER ENFORCEMENT** Forward-looking statements require qualifiers. | Fails | Passes | |-------|--------| | "Revenue will be $50M by Q4" | "Revenue is **projected** to be $50M by Q4" | | "The feature will reduce churn" | "The feature is **expected** to reduce churn" | **Fix:** Add "projected", "estimated", "expected", "designed to", "intended to" --- **3. ASSUMPTION VISIBILITY** Implicit assumptions that affect interpretation must be explicit. | Fails | Passes | |-------|--------| | "The system scales linearly" | "The system scales linearly [assuming <1000 concurrent users]" | | "Response time is 50ms" | "Response time is 50ms [under standard load conditions]" | **Fix:** Add bracketed conditions: "[assuming X]", "[under conditions Y]", "[when Z]" --- **4. AUTHORITATIVE-LOOKING UNVALIDATED DATA** Tables with specific percentages and checkmarks look like measured data. **Red flag:** Tables with specific numbers (89%, 95%, 100%) without sources **Fix:** Add "(guess)", "(est.)", "?" to numbers. Add explicit warning: "PROJECTED VALUES - NOT MEASURED" --- ### Data Quality Checks (Complementary: Points 5-7) **5. DATA CONSISTENCY** Scan for conflicting numbers, dates, or facts within the document. **Red flag:** "500 users" in one section, "750 users" in another **Fix:** Reconcile conflicts or explicitly note the discrepancy with explanation. --- **6. IMPLICIT CAUSATION** Claims that imply causation without evidence. **Red flag:** "Shorter prompts improve response quality" (plausible but unproven) **Fix:** Reframe as hypothesis: "Shorter prompts MAY improve response quality (hypothesis, not validated)" --- **7. FUTURE STATE AS PRESENT** Describing planned/hoped outcomes as if already achieved. **Red flag:** "The system processes 10,000 requests per second" (when it hasn't been built) **Fix:** Use future/conditional: "The system is DESIGNED TO process..." or "TARGET: 10,000 rps" --- ### Verification Routing (Points 8-9) **8. TEMPORAL COHERENCE** Document dates and timestamps must be internally consistent and plausible. | Fails | Passes | |-------|--------| | "Last Updated: December 2024" (when current is 2026) | "Last Updated: January 2026" | | v1.0.0 dated 2024-12-23, v1.1.0 dated 2024-12-20 | Versions in chronological order | **Sub-checks:** 1. Document date vs current date 2. Internal chronology (versions, events in order) 3. Reference freshness ("current", "now", "today" claims) **Fix:** Update dates, add "as of [date]" qualifiers, flag stale claims --- **9. EXTERNALLY VERIFIABLE CLAIMS** Specific numbers that could be fact-checked should be flagged for verification. | Type | Example | Risk | |------|---------|------| | Pricing | "Costs ~$0.005 per call" | API pricing changes | | Statistics | "Papers average 15-30 equations" | May be wildly off | | Rates/ratios | "40% of researchers use X" | Needs citation | | Competitor claims | "No competitor offers Y" | May be outdated | **Fix options:** 1. Add source with date 2. Add uncertainty marker 3. Route to HITL or external search 4. Generalize ("low cost" instead of "$0.005") --- ## The Verification Hierarchy ``` Claim Extracted --> Does Source of Truth Exist? | +---------------+---------------+ YES NO | | Tier 1: Automated Tier 2: HITL Consistency & Verification Two-Round Verification | | PASS / BLOCK Round A → Round B → APPROVE / REJECT ``` ### Tier 1: Automated Verification **A. Internal Consistency** - Figure vs. Text contradictions - Abstract vs. Body mismatches - Table vs. Prose conflicts - Numerical consistency **B. External Verification (Extension Interface)** - User-provided connectors to structured sources - Financial systems, Git commits, CRM, etc. ### Tier 2: Two-Round HITL Verification — MANDATORY **Round A: Derived Data Confirmation** - Claims from sources found in session - Human confirms interpretation, not truth **Round B: True HITL Verification** - Claims needing actual verification - No source found, human's own data, extrapolations --- ## CGD Output Format When producing a Clarity-Gated Document, use this format per [CLARITY_GATE_FORMAT_SPEC.md](../../docs/CLARITY_GATE_FORMAT_SPEC.md) v2.1: ```yaml --- clarity-gate-version: 2.1 processed-date: 2026-01-12 processed-by: Claude + Human Review clarity-status: CLEAR hitl-status: REVIEWED hitl-pending-count: 0 points-passed: 1-9 rag-ingestable: true # computed by validator - do not set manually document-sha256: 7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730 hitl-claims: - id: claim-75fb137a text: "Revenue projection is $50M" value: "$50M" source: "Q3 planning doc" location: "revenue-projections/1" round: B confirmed-by: Francesco confirmed-date: 2026-01-12 --- # Document Title [Document body with epistemic markers applied] Claims like "Revenue will be $50M" become "Revenue is **projected** to be $50M *(unverified projection)*" --- ## HITL Verification Record ### Round A: Derived Data Confirmation - Claim 1 (source) ✓ - Claim 2 (source) ✓ ### Round B: True HITL Verification | # | Claim | Status | Verified By | Date | |---|-------|--------|-------------|------| | 1 | [claim] | ✓ Confirmed | [name] | [date] | <!-- CLARITY_GATE_END --> Clarity Gate: CLEAR | REVIEWED ``` **Required CGD Elements (per spec):** - YAML frontmatter with all required fields: - `clarity-gate-version` — Tool version (no "v" prefix) - `processed-date` — YYYY-MM-DD format - `processed-by` — Processor name - `clarity-status` — CLEAR or UNCLEAR - `hitl-status` — PENDING, REVIEWED, or REVIEWED_WITH_EXCEPTIONS - `hitl-pending-count` — Integer ≥ 0 - `points-passed` — e.g., `1-9` or `1-4,7,9` - `hitl-claims` — List of verified claims (may be empty `[]`) - End marker (HTML comment + status line): ``` <!-- CLARITY_GATE_END --> Clarity Gate: <clarity-status> | <hitl-status> ``` - HITL verification record (if status is REVIEWED) **Optional/Computed Fields:** - `rag-ingestable` — **Computed by validators**, not manually set. Shows `true` only when `CLEAR | REVIEWED` with no exclusion blocks. - `document-sha256` — Required. 64-char lowercase hex hash for integrity verification. See spec §2 for computation rules. - `exclusions-coverage` — Optional. Fraction of body inside exclusion blocks (0.0–1.0). **Escape Mechanism:** To write about markers like `*(estimated)*` without triggering parsing, wrap in backticks: `` `*(estimated)*` `` ### Claim Completion Status (v2.1) Claim verification status is determined by field **presence**, not an explicit status field: | State | `confirmed-by` | `confirmed-date` | Meaning | |-------|----------------|------------------|----------| | **PENDING** | absent | absent | Awaiting human verification | | **VERIFIED** | present | present | Human has confirmed | | *(invalid)* | present | absent | W-HC01: partial fields | | *(invalid)* | absent | present | W-HC01: partial fields | **Why no explicit status field?** Field presence is self-enforcing—you can't accidentally set status without providing who/when. ### Source Field Semantics (v2.1) The `source` field meaning changes based on claim state: | State | `source` Contains | Example | |-------|-------------------|----------| | **PENDING** | Where to verify (actionable) | `"Check Q3 planning doc"` | | **VERIFIED** | What was found (evidence) | `"Q3 planning doc, page 12"` | **Vague source detection (W-HC02):** Sources like `"industry reports"`, `"research"`, `"TBD"` trigger warnings. ### Claim ID Format (v2.1) **General pattern:** `claim-[a-z0-9._-]{1,64}` (alphanumeric, dots, underscores, hyphens) | Approach | Pattern | Example | Use Case | |----------|---------|---------|----------| | **Hash-based** (preferred) | `claim-[a-f0-9]{8,}` | `claim-75fb137a` | Deterministic, collision-resistant | | **Sequential** | `claim-[0-9]+` | `claim-1`, `claim-2` | Simple documents | | **Semantic** | `claim-[a-z0-9-]+` | `claim-revenue-q3` | Human-friendly | **Collision probability:** At 1,000 claims with 8-char hex IDs: ~0.012%. For >1,000 claims, use 12+ hex characters. **Recommendation:** Use hash-based IDs generated by `scripts/claim_id.py` for consistency and collision resistance. --- ## Exclusion Blocks When content cannot be resolved (no SME available, legacy prose, etc.), mark it as excluded rather than leaving it ambiguous: ```markdown <!-- CG-EXCLUSION:BEGIN id=auth-legacy-1 --> Legacy authentication details that require SME review... <!-- CG-EXCLUSION:END id=auth-legacy-1 --> ``` **Rules:** - IDs must match: `[A-Za-z0-9][A-Za-z0-9._-]{0,63}` - No nesting or overlapping blocks - Each ID used only once - Requires `hitl-status: REVIEWED_WITH_EXCEPTIONS` - Must document `exceptions-reason` and `exceptions-ids` in frontmatter **Important:** Documents with exclusion blocks are **not RAG-ingestable**. They're rejected entirely (no partial ingestion). See [CLARITY_GATE_FORMAT_SPEC.md §4](../../docs/CLARITY_GATE_FORMAT_SPEC.md) for complete rules. --- ## SOT Validation When validating a Source of Truth file, the skill checks both **format compliance** (per [CLARITY_GATE_FORMAT_SPEC.md](../../docs/CLARITY_GATE_FORMAT_SPEC.md)) and **content quality** (the 9 points). ### Format Compliance (Structural Rules) SOT documents are CGDs with a `tier:` block. They require a `## Verified Claims` section with a valid table. | Code | Check | Severity | |------|-------|----------| | E-TB01 | No `## Verified Claims` section | ERROR | | E-TB02 | Table has no data rows | ERROR | | E-TB03 | Required columns missing (Claim, Value, Source, Verified) | ERROR | | E-TB04 | Column order wrong (Claim not first or Verified not last) | ERROR | | E-TB05 | Empty cell in required column | ERROR | | E-TB06 | Invalid date format in Verified column | ERROR | | E-TB07 | Verified date in future (beyond 24h grace) | ERROR | ### Content Quality (9 Points) The 9 Verification Points apply to SOT content: | Point | SOT Application | |-------|-----------------| | 1-4 | Check claims in `## Verified Claims` are actually verified | | 5 | Check for conflicting values across tables | | 6 | Check claims don't imply unsupported causation | | 7 | Check table doesn't state futures as present | | 8 | Check dates are chronologically consistent | | 9 | Flag specific numbers for external check | ### SOT-Specific Requirements - **Tier block required:** SOT is a CGD with `tier:` block containing `level`, `owner`, `version`, `promoted-date`, `promoted-by` - **Structured claims table:** `## Verified Claims` section with columns: Claim, Value, Source, Verified - **Table outside exclusions:** The verified claims table must NOT be inside an exclusion block - **Staleness markers:** Use `[STABLE]`, `[CHECK]`, `[VOLATILE]`, `[SNAPSHOT]` in content - `[STABLE]` — Safe to cite without rechecking - `[CHECK]` — Verify before citing - `[VOLATILE]` — Changes frequently; always verify - `[SNAPSHOT]` — Point-in-time data; include date when citing --- ## Output Format After running Clarity Gate, report: ``` ## Clarity Gate Results **Document:** [filename] **Issues Found:** [number] ### Critical (will cause hallucination) - [issue + location + fix] ### Warning (could cause equivocation) - [issue + location + fix] ### Temporal (date/time issues) - [issue + location + fix] ### Externally Verifiable Claims | # | Claim | Type | Suggested Verification | |---|-------|------|------------------------| | 1 | [claim] | Pricing | [where to verify] | --- ## Round A: Derived Data Confirmation - [claim] ([source]) Reply "confirmed" or flag any I misread. --- ## Round B: HITL Verification Required | # | Claim | Why HITL Needed | Human Confirms | |---|-------|-----------------|----------------| | 1 | [claim] | [reason] | [ ] True / [ ] False | --- **Would you like me to produce an annotated CGD version?** --- **Verdict:** PENDING CONFIRMATION ``` --- ## Severity Levels | Level | Definition | Action | |-------|------------|--------| | **CRITICAL** | LLM will likely treat hypothesis as fact | Must fix before use | | **WARNING** | LLM might misinterpret | Should fix | | **TEMPORAL** | Date/time inconsistency detected | Verify and update | | **VERIFIABLE** | Specific claim that could be fact-checked | Route to HITL or external search | | **ROUND A** | Derived from witnessed source | Quick confirmation | | **ROUND B** | Requires true verification | Cannot pass without confirmation | | **PASS** | Clearly marked, no ambiguity, verified | No action needed | --- ## Quick Scan Checklist | Pattern | Action | |---------|--------| | Specific percentages (89%, 73%) | Add source or mark as estimate | | Comparison tables | Add "PROJECTED" header | | "Achieves", "delivers", "provides" | Use "designed to", "intended to" if not validated | | Checkmarks | Verify these are confirmed | | "100%" anything | Almost always needs qualification | | "Last Updated: [date]" | Check against current date | | Version numbers with dates | Verify chronological order | | "$X.XX" or "~$X" (pricing) | Flag for external verification | | "averages", "typically" | Flag for source/citation | | Competitor capability claims | Flag for external verification | --- ## What This Skill Does NOT Do - Does not classify document types (use Stream Coding for that) - Does not restructure documents - Does not add deep links or references - Does not evaluate writing quality - **Does not check factual accuracy autonomously** (requires HITL) --- ## Related Projects | Project | Purpose | URL | |---------|---------|-----| | Source of Truth Creator | Create epistemically calibrated docs | github.com/frmoretto/source-of-truth-creator | | Stream Coding | Documentation-first methodology | github.com/frmoretto/stream-coding | | ArXiParse | Scientific paper verification | arxiparse.org | --- ## Changelog ### v2.1.3 (2026-03-02) - **FIXED:** `document_hash.py` now implements full FORMAT_SPEC §2.1-2.4 compliance - **FIXED:** Fence-aware end marker detection (Quine Protection per §2.3/§8.5) - **FIXED:** All 4 deployment copies converged to single canonical implementation - **ADDED:** `canonicalize()` function: trailing whitespace stripping, newline collapsing, NFC normalization - **ADDED:** YAML-aware `document-sha256` removal with multiline continuation support (§2.2) - **ADDED:** Fence-tracking test vectors (7 new tests, 15 total) ### v2.1.0 (2026-01-27) - **ADDED:** Claim Completion Status semantics (PENDING/VERIFIED by field presence) - **ADDED:** Source Field Semantics (actionable vs. what-was-found) - **ADDED:** Claim ID Format guidance with collision analysis - **ADDED:** Body Structure Requirements (HITL Verification Record mandatory when claims exist) - **ADDED:** New validation codes: E-ST10, W-ST11, W-HC01, W-HC02, E-SC06 (FORMAT_SPEC §1.2-1.3) - **ADDED:** Bundled scripts: `claim_id.py`, `document_hash.py` - **UPDATED:** References to FORMAT_SPEC v2.1 - **UPDATED:** CGD output example to version 2.1 ### v2.0.0 (2026-01-13) - **ADDED:** agentskills.io compliant YAML frontmatter - **ADDED:** Clarity Gate Format Specification v2.0 compliance (unified CGD/SOT) - **ADDED:** SOT validation support with E-TB* error codes - **ADDED:** Validation rules mapping (9 points → rule codes) - **ADDED:** CGD output format template with `<!-- CLARITY_GATE_END -->` markers - **ADDED:** Quine Protection note (§2.3 fence-aware marker detection) - **ADDED:** Redacted Export feature (§8.11) - **UPDATED:** `hitl-claims` format to v2.0 schema (id, text, value, source, location, round) - **UPDATED:** End marker format to HTML comment style - **UPDATED:** Unified format spec v2.0 (single `.cgd.md` extension) - **RESTRUCTURED:** For multi-platform skill discovery ### v1.6 (2025-12-31) - Added Two-Round HITL verification system - Round A: Derived Data Confirmation - Round B: True HITL Verification ### v1.5 (2025-12-28) - Added Point 8: Temporal Coherence - Added Point 9: Externally Verifiable Claims ### v1.4 (2025-12-23) - Added CGD annotation output mode ### v1.3 (2025-12-21) - Restructured points into Epistemic (1-4) and Data Quality (5-7) ### v1.2 (2025-12-21) - Added Source of Truth request step ### v1.1 (2025-12-21) - Added HITL Fact Verification (mandatory) ### v1.0 (2025-11) - Initial release with 6-point verification --- **Version:** 2.1.3 **Spec Version:** 2.1 **Author:** Francesco Marinoni Moretto **License:** CC-BY-4.0

path: scripts/claim_id.py
content: #!/usr/bin/env python3
"""
Clarity Gate: Claim ID Computation

Reference implementation per FORMAT_SPEC §1.3.4 (RFC-001).
Produces stable, hash-based claim IDs for HITL tracking.

Usage:
    python claim_id.py "Base price is $99/mo" "api-pricing/1"
    # Output: claim-75fb137a

Test Vectors (FORMAT_SPEC §18.2):
    claim_id("Base price is $99/mo", "api-pricing/1") == "claim-75fb137a"
    claim_id("The API supports GraphQL", "features/1") == "claim-eb357742"

Normalization (per RFC-001 §3.4):
    - Text: strip outer whitespace, collapse internal whitespace to single space
    - Location: strip outer whitespace
    
    This ensures stable IDs regardless of formatting variations:
    "Price is  $99" (2 spaces) → same hash as "Price is $99" (1 space)
"""

import hashlib
import sys


def normalize_text(text: str) -> str:
    """
    Normalize claim text for consistent hashing.
    
    Per RFC-001 §3.4:
    - Strip leading/trailing whitespace
    - Collapse multiple internal spaces to single space
    """
    return ' '.join(text.strip().split())


def claim_id(text: str, location: str) -> str:
    """
    Compute a stable claim ID from claim text and location.
    
    Args:
        text: The full claim text (e.g., "Base price is $99/mo")
        location: The heading_slug/ordinal (e.g., "api-pricing/1")
    
    Returns:
        Claim ID in format "claim-XXXXXXXX" (8-char hex hash)
    
    Algorithm:
        1. Normalize text (strip + collapse whitespace)
        2. Strip location whitespace
        3. Concatenate with pipe delimiter
        4. SHA-256 hash the UTF-8 encoded payload
        5. Take first 8 hex characters
        6. Prefix with "claim-"
    """
    normalized_text = normalize_text(text)
    normalized_location = location.strip()
    
    payload = f"{normalized_text}|{normalized_location}"
    hash_hex = hashlib.sha256(payload.encode('utf-8')).hexdigest()
    return f"claim-{hash_hex[:8]}"


def main():
    if len(sys.argv) == 3:
        text = sys.argv[1]
        location = sys.argv[2]
        print(claim_id(text, location))
    elif len(sys.argv) == 2 and sys.argv[1] == "--test":
        # Run test vectors
        print("=== Test Vectors ===")
        tests = [
            ("Base price is $99/mo", "api-pricing/1", "claim-75fb137a"),
            ("The API supports GraphQL", "features/1", "claim-eb357742"),
        ]
        all_pass = True
        for text, location, expected in tests:
            result = claim_id(text, location)
            status = "PASS" if result == expected else "FAIL"
            if result != expected:
                all_pass = False
            print(f"{status} claim_id({text!r}, {location!r})")
            print(f"    Expected: {expected}")
            print(f"    Got:      {result}")
        
        print("\n=== Normalization Tests ===")
        # These should all produce the same hash
        norm_tests = [
            ("Price is $99", "loc/1"),
            ("Price is  $99", "loc/1"),      # double space
            ("  Price is $99  ", "loc/1"),   # outer spaces
            ("Price is $99", "  loc/1  "),   # location spaces
        ]
        base_result = claim_id(norm_tests[0][0], norm_tests[0][1])
        print(f"Base: {base_result}")
        for text, location in norm_tests[1:]:
            result = claim_id(text, location)
            status = "PASS" if result == base_result else "FAIL"
            if result != base_result:
                all_pass = False
            print(f"{status} claim_id({text!r}, {location!r}) == base")
        
        print()
        sys.exit(0 if all_pass else 1)
    else:
        print("Usage: python claim_id.py <text> <location>")
        print("       python claim_id.py --test")
        print()
        print("Example:")
        print('  python claim_id.py "Base price is $99/mo" "api-pricing/1"')
        print("  # Output: claim-75fb137a")
        sys.exit(1)


if __name__ == "__main__":
    main()


path: scripts/document_hash.py
content: #!/usr/bin/env python3
"""
Clarity Gate: Document Hash Computation

Reference implementation per FORMAT_SPEC §2.1-2.4, §2.3 (RFC-001).
Computes SHA-256 hash excluding the document-sha256 line itself.

Usage:
    python document_hash.py path/to/file.cgd.md
    python document_hash.py --verify path/to/file.cgd.md
    python document_hash.py --test

Algorithm per FORMAT_SPEC:
    0. Pre-normalize: BOM removal, CRLF/CR to LF
    1. Extract hash window: content between '---\\n' and '<!-- CLARITY_GATE_END -->'
       - End marker detection uses fence-aware scanning (§2.3 Quine Protection)
    2. Exclude document-sha256 line(s) from YAML frontmatter only
       - Multiline YAML continuation handling (§2.2)
    3. Canonicalize per §2.4:
       - Strip trailing whitespace per line
       - Collapse 3+ consecutive newlines to 2
       - Normalize final newline (exactly 1 LF)
       - UTF-8 NFC normalization
    4. Compute SHA-256
"""

import hashlib
import re
import sys
import unicodedata


def canonicalize(text: str) -> str:
    """
    Canonicalize content for consistent hashing across platforms.

    Per FORMAT_SPEC §2.4:
    1. Trailing whitespace: Remove per line
    2. Consecutive newlines: Collapse 3+ to 2
    3. Final newline: Exactly one trailing LF
    4. Encoding: UTF-8 NFC normalization
    """
    # 1. Strip trailing whitespace per line
    lines = text.split('\n')
    lines = [line.rstrip() for line in lines]
    text = '\n'.join(lines)

    # 2. Collapse 3+ consecutive newlines to 2
    while '\n\n\n' in text:
        text = text.replace('\n\n\n', '\n\n')

    # 3. Normalize trailing newline (exactly 1)
    text = text.rstrip('\n') + '\n'

    # 4. UTF-8 NFC normalization
    text = unicodedata.normalize('NFC', text)

    return text


# Regex for fence opener/closer: 0-3 leading spaces, then 3+ backticks or tildes
_FENCE_RE = re.compile(r'^( {0,3})(`{3,}|~{3,})')

END_MARKER = '<!-- CLARITY_GATE_END -->'


def find_end_marker(text: str) -> int:
    """
    Find position of first <!-- CLARITY_GATE_END --> outside fenced code blocks.

    Per FORMAT_SPEC §2.3 (Quine Protection) and §8.5 fence-tracking:
    - Fence opens on line starting with 3+ backticks/tildes (after 0-3 spaces)
    - Fence closes on line with same character, equal or greater count
    - Lines indented 4+ spaces do NOT open/close fences
    - Info strings after opener are ignored

    Returns: character offset of the marker.
    Raises: ValueError if no valid marker found.
    """
    in_fence = False
    fence_char = ''
    fence_count = 0
    offset = 0

    for line in text.split('\n'):
        if not in_fence:
            marker_pos = line.find(END_MARKER)
            if marker_pos != -1:
                return offset + marker_pos

        m = _FENCE_RE.match(line)
        if m:
            char = m.group(2)[0]
            count = len(m.group(2))
            if not in_fence:
                in_fence = True
                fence_char = char
                fence_count = count
            elif char == fence_char and count >= fence_count:
                in_fence = False

        offset += len(line) + 1  # +1 for the \n consumed by split

    raise ValueError("No <!-- CLARITY_GATE_END --> found outside fenced code blocks")


def compute_hash(filepath: str) -> str:
    """
    Compute SHA-256 hash of document per FORMAT_SPEC §2.2-2.4.

    Algorithm:
        0. Pre-normalize for boundary detection (BOM, CRLF)
        1. Extract content between opening '---\n' and '<!-- CLARITY_GATE_END -->'
        2. Remove document-sha256 line(s) from YAML frontmatter ONLY
           (including multiline continuations)
        3. Canonicalize per §2.4
        4. Compute SHA-256
    """
    with open(filepath, 'r', encoding='utf-8') as f:
        file_content = f.read()

    # 0. Pre-normalize for boundary detection
    working = file_content
    if working.startswith('\ufeff'):
        working = working[1:]
    working = working.replace('\r\n', '\n').replace('\r', '\n')

    # 1. Extract content between opening YAML delimiter and end marker
    #    End marker uses fence-aware detection per §2.3 (Quine Protection)
    try:
        start = working.index('---\n') + len('---\n')
        end = find_end_marker(working)
        hashable = working[start:end]
    except ValueError as e:
        print(f"ERROR: Invalid CGD format - {e}")
        sys.exit(1)

    # 2. Remove document-sha256 line(s) - YAML frontmatter only
    lines = hashable.split('\n')
    filtered = []
    skip_multiline = False
    hash_indent = 0
    in_frontmatter = True

    for line in lines:
        # Detect end of YAML frontmatter
        if in_frontmatter and line.strip() == '---':
            in_frontmatter = False

        # Check if this is the hash line
        if in_frontmatter and re.match(r'^\s*document-sha256:', line):
            skip_multiline = True
            hash_indent = len(line) - len(line.lstrip())
            continue

        # If we're skipping multiline, check if this is a continuation
        if skip_multiline:
            current_indent = len(line) - len(line.lstrip())
            if in_frontmatter and current_indent > hash_indent:
                continue
            skip_multiline = False

        filtered.append(line)

    hashable = '\n'.join(filtered)

    # 3. Canonicalize
    hashable = canonicalize(hashable)

    # 4. Compute
    return hashlib.sha256(hashable.encode('utf-8')).hexdigest()


def verify(filepath: str) -> bool:
    """
    Verify document hash matches stored value.

    Returns True if hash matches, False otherwise.
    """
    with open(filepath, 'r', encoding='utf-8') as f:
        content = f.read()

    # Normalize for consistent matching
    if content.startswith('\ufeff'):
        content = content[1:]
    content = content.replace('\r\n', '\n').replace('\r', '\n')

    # Extract stored hash
    match = re.search(r'^\s*document-sha256:\s*["\']?([a-f0-9]{64})["\']?', content, re.MULTILINE)
    if not match:
        print("FAIL: No document-sha256 found")
        return False

    stored = match.group(1)
    computed = compute_hash(filepath)

    if stored == computed:
        print(f"PASS: Hash verified: {computed}")
        return True
    else:
        print(f"FAIL: Hash mismatch")
        print(f"  Stored:   {stored}")
        print(f"  Computed: {computed}")
        return False


def run_tests():
    """Run canonicalization and edge case tests."""
    print("=== Canonicalization Tests ===")

    # Test 1: BOM removal (happens in pre-normalization, not canonicalize)
    # Note: BOM is removed in compute_hash step 0, not in canonicalize
    with_bom = '\ufeff# Test'
    without_bom = '# Test'
    # After BOM removal in pre-normalization:
    assert with_bom.lstrip('\ufeff') == without_bom, "BOM removal in pre-normalization"
    print("PASS: BOM removal (pre-normalization)")

    # Test 2: Trailing whitespace removal
    with_trailing = "line1  \nline2\t\nline3"
    without_trailing = "line1\nline2\nline3"
    assert canonicalize(with_trailing) == canonicalize(without_trailing), "Trailing whitespace should be stripped"
    print("PASS: Trailing whitespace removal")

    # Test 3: Newline collapsing (3+ → 2)
    multiple_newlines = "para1\n\n\n\npara2"
    collapsed = "para1\n\npara2"
    assert canonicalize(multiple_newlines) == collapsed + '\n', "3+ newlines should collapse to 2"
    print("PASS: Newline collapsing")

    # Test 4: Final newline normalization
    no_trailing = "content"
    one_trailing = "content\n"
    two_trailing = "content\n\n"
    assert canonicalize(no_trailing) == "content\n", "Missing final newline should be added"
    assert canonicalize(one_trailing) == "content\n", "Single final newline should be preserved"
    assert canonicalize(two_trailing) == "content\n", "Multiple final newlines should collapse to 1"
    print("PASS: Final newline normalization")

    # Test 5: NFC normalization
    # é as single codepoint (U+00E9) vs e + combining acute (U+0065 U+0301)
    nfc = '\u00e9'  # NFC form
    nfd = '\u0065\u0301'  # NFD form
    assert canonicalize(nfc) == canonicalize(nfd), "NFC normalization should make equivalent"
    print("PASS: UTF-8 NFC normalization")

    # Test 6: Preserve tabs and leading whitespace
    with_tabs = "line1\n\tindented\n  spaces"
    canonical = canonicalize(with_tabs)
    assert '\t' in canonical, "Tabs should be preserved"
    assert '  spaces' in canonical, "Leading whitespace should be preserved"
    print("PASS: Preserve tabs and leading whitespace")

    print("\n=== Line Ending Tests ===")

    # Test 7: CRLF normalization (in pre-processing)
    crlf = "line1\r\nline2\r\n"
    lf = "line1\nline2\n"
    # Note: CRLF normalization happens in compute_hash step 0, not canonicalize
    assert canonicalize(crlf.replace('\r\n', '\n')) == canonicalize(lf), "CRLF should normalize to LF"
    print("PASS: CRLF to LF")

    # Test 8: CR normalization
    cr = "line1\rline2\r"
    assert canonicalize(cr.replace('\r', '\n')) == canonicalize(lf), "CR should normalize to LF"
    print("PASS: CR to LF")

    print("\n=== Fence-Aware End Marker Tests (§2.3 Quine Protection) ===")

    # Test 9: Simple marker detection (no fences)
    simple = "some content\n<!-- CLARITY_GATE_END -->\nafter"
    assert find_end_marker(simple) == simple.index(END_MARKER), \
        "Simple end marker detection"
    print("PASS: Simple end marker detection")

    # Test 10: Marker inside backtick fence should be skipped
    fenced = "before\n```\n<!-- CLARITY_GATE_END -->\n```\n<!-- CLARITY_GATE_END -->\nafter"
    expected = fenced.rfind(END_MARKER)
    assert find_end_marker(fenced) == expected, \
        "Marker inside backtick fence should be skipped"
    print("PASS: Marker inside backtick fence skipped")

    # Test 11: Marker inside tilde fence should be skipped
    tilde = "before\n~~~\n<!-- CLARITY_GATE_END -->\n~~~\n<!-- CLARITY_GATE_END -->\nafter"
    expected = tilde.rfind(END_MARKER)
    assert find_end_marker(tilde) == expected, \
        "Marker inside tilde fence should be skipped"
    print("PASS: Marker inside tilde fence skipped")

    # Test 12: Longer fence — ``` does NOT close ````
    long_fence = "````\n<!-- CLARITY_GATE_END -->\n```\n<!-- CLARITY_GATE_END -->\n````\n<!-- CLARITY_GATE_END -->"
    expected = long_fence.rfind(END_MARKER)
    assert find_end_marker(long_fence) == expected, \
        "Shorter fence delimiter should not close longer fence"
    print("PASS: Fence length tracking")

    # Test 13: 4+ space indent does NOT open a fence
    indented = "    ```\n<!-- CLARITY_GATE_END -->\nafter"
    expected = indented.index(END_MARKER)
    assert find_end_marker(indented) == expected, \
        "4+ space indented line should not open fence"
    print("PASS: Indented code block ignored")

    # Test 14: Missing marker raises ValueError
    try:
        find_end_marker("no marker here")
        assert False, "Should have raised ValueError"
    except ValueError:
        pass
    print("PASS: Missing marker raises ValueError")

    # Test 15: Backtick fence not closed by tilde fence
    mixed = "```\n<!-- CLARITY_GATE_END -->\n~~~\n<!-- CLARITY_GATE_END -->\n```\n<!-- CLARITY_GATE_END -->"
    expected = mixed.rfind(END_MARKER)
    assert find_end_marker(mixed) == expected, \
        "Tilde should not close backtick fence"
    print("PASS: Mixed fence characters respected")

    print("\nPASS: All tests passed")


def main():
    if len(sys.argv) == 2 and sys.argv[1] not in ("--verify", "--test"):
        print(compute_hash(sys.argv[1]))
    elif len(sys.argv) == 3 and sys.argv[1] == "--verify":
        sys.exit(0 if verify(sys.argv[2]) else 1)
    elif len(sys.argv) == 2 and sys.argv[1] == "--test":
        run_tests()
    else:
        print("Usage: document_hash.py <file>")
        print("       document_hash.py --verify <file>")
        print("       document_hash.py --test")
        print()
        print("Examples:")
        print("  document_hash.py my-doc.cgd.md")
        print("  document_hash.py --verify my-doc.cgd.md")
        print("  document_hash.py --test")
        sys.exit(1)


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)