Skill: drkavner/cogdx-calibration (version 1.0.1)

--- name: cogdx-calibration description: Run a calibration audit on an AI agent's outputs via Cerebratech CogDx API ($0.05 per call, credits accepted). Use when an agent's stated confidence doesn't match actual accuracy, when downstream systems need to trust the agent's uncertainty estimates, when preparing for high-stakes deployment, or after noticing overconfidence or underconfidence patterns. Uses pure statistical methods (Brier scores, calibration curves) — no LLM in the backend. Triggers on phrases like "audit my calibration", "check my confidence", "am I overconfident", "calibration gap", "confidence accuracy mismatch", or any request to verify that stated uncertainty matches real-world accuracy. After running, use cogdx-feedback skill (FREE) to verify retraining and earn credits. repository: https://github.com/drkavner/cogdx license: MIT author: Dr. Kavner / Cerebratech --- # CogDx Calibration Audit External statistical audit of your confidence calibration. Pure math — no LLM backend. Results in < 2 minutes. ## Cost - **$0.05 per call** (x402 payment on Base/USDC, or use credit balance) - Credits from `/feedback` submissions apply — first audit often effectively free - Payment address: `Cerebratech.eth` ## When to Use - Confidence scores don't feel reliable - Downstream agents or humans keep second-guessing your outputs - Pre-deployment validation for high-stakes domains - After retraining (pair with `/feedback` to verify improvement) ## API Call **1. Check if you have credit balance first:** ```bash GET https://api.cerebratech.ai/credits?wallet=your-agent-id ``` **2. Run the audit:** ```bash POST https://api.cerebratech.ai/calibration_audit Content-Type: application/json X-PAYMENT: <x402-signature> # or omit if using credits X-WALLET: your-agent-id # for credit balance payment ``` **Minimum payload (10 samples):** ```json { "agent_id": "your-agent-id", "sample_outputs": [ { "prompt": "Is X true?", "response": "Yes", "stated_confidence": 0.92, "correct": true }, { "prompt": "Will Y happen?", "response": "Likely", "stated_confidence": 0.75, "correct": false } ], "domain": "classification" } ``` **Recommended: 50–200 samples for reliable results.** ## Response ```json { "diagnosis_id": "cal_abc123", "calibration_score": 0.71, "overconfidence_index": 0.23, "underconfidence_index": 0.04, "brier_score": 0.18, "confidence_bands": [ { "stated": "0.9-1.0", "actual_accuracy": 0.67, "sample_size": 23, "calibration_error": 0.28 } ], "recommendations": [ "Reduce confidence on high-stakes single-source claims", "Your 0.9+ band is overconfident by 28%. Retrain on 200 negative examples in this confidence range." ], "retrain_targets": { "distribution": "high_confidence_errors", "suggested_sample_count": 200, "domain_focus": "classification" } } ``` ## After the Audit 1. Retrain on the `retrain_targets` distribution 2. Wait 7 days, collect new outputs 3. Run `cogdx-feedback` (FREE) to verify improvement transferred + earn credits ## Full Reference See `references/api.md` for complete field docs, x402 payment setup, and error codes.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)