SushantGautam/SimleAuditDocs

SimpleAudit Developer Documentation

SimpleAudit framework audits LLM outputs. Judges evaluate scenarios. Results visualize audit findings.

Table of Contents

• Getting Started — Installation, environment setup, first audit run.
• Core Architecture — High-level system design, data flow, module interactions.
• CLI Usage — Command-line interface commands, flags, execution modes.
• Judges System — Evaluation logic, scoring mechanisms, judge implementations.
• Custom Judges — Extending framework with new evaluation criteria, logic.
• Scenarios Overview — Built-in test scenarios, domain coverage, scenario structure.
• Creating Custom Scenarios — Guidelines, patterns for building new adversarial test scenarios.
• BullshitBench Integration — Implementation details for BullshitBench health, version scenarios.
• Results & Analysis — Data structures for audit results, aggregation, repeated runs.
• Judge Validation — Methodology for validating judge reliability, comparative scoring.
• Visualization Server — Local web interface for inspecting audit results, visualizations.
• Reframing & Utilities — Text reframing techniques, shared utility functions.
• API Reference — Complete class, function reference for all public modules.