npm package: @sovereign-labs/improve
Description: Self-improving verification pipeline. Diagnose failures, generate fix candidates, validate in isolation, reject overfitting. Repair with proof.
Latest version: 0.1.0

# @sovereign-labs/improve

Self-improving verification pipeline. Diagnose failures, generate fix candidates, validate in isolation, reject overfitting. **Repair with proof.**

Works with any test runner implementing the `TestSurface` interface — not coupled to `@sovereign-labs/verify`, though that's the primary consumer.

## Pipeline

```
baseline → bundle → triage → diagnose → generate → validate → rank → holdout → verdict
```

1. **Baseline** — Run all scenarios, identify dirty (failing) ones
2. **Bundle** — Group violations by root cause into evidence bundles
3. **Triage** — Classify confidence: `mechanical` (pattern-match), `heuristic`, or `needs_llm`
4. **Diagnose** — LLM root-cause analysis (skipped for mechanical triage)
5. **Generate** — LLM produces N fix candidates (search/replace edits)
6. **Validate** — Each candidate tested in isolated subprocess copy
7. **Rank** — Score = improvements - regressions - line penalty
8. **Holdout** — Best candidate re-tested against withheld scenarios (overfitting detection)
9. **Verdict** — `accepted`, `rejected_regression`, `rejected_overfitting`, `rejected_no_fix`, `skipped_*`

## Install

```bash
npm install @sovereign-labs/improve
# or
bun add @sovereign-labs/improve
```

## CLI Usage

```bash
# With Gemini
improve --app-dir ./my-package --llm gemini --api-key $GEMINI_KEY

# With local Ollama
improve --app-dir ./my-package --llm ollama --ollama-model qwen3:4b

# With Claude (domain-aware prompts)
improve --app-dir ./my-package --llm claude --api-key $ANTHROPIC_API_KEY

# Dry run (generate candidates, skip validation)
improve --app-dir ./my-package --llm gemini --api-key $GEMINI_KEY --dry-run

# Specific scenario families only
improve --app-dir ./my-package --llm gemini --api-key $GEMINI_KEY --families grounding,constraints
```

### CLI Options

| Option | Description | Default |
|--------|-------------|---------|
| `--app-dir <path>` | Package directory to improve | `.` |
| `--self-test <script>` | Self-test script path (relative to app-dir) | `scripts/self-test.ts` |
| `--llm <provider>` | LLM provider: `gemini`, `anthropic`, `claude`, `claude-code`, `ollama`, `none` | `none` |
| `--api-key <key>` | API key for cloud providers | — |
| `--ollama-model <model>` | Ollama model name | `qwen3:4b` |
| `--ollama-host <url>` | Ollama host URL | `http://localhost:11434` |
| `--claude-model <model>` | Claude model name | `claude-sonnet-4-20250514` |
| `--max-candidates <n>` | Fix candidates per bundle | `3` |
| `--max-lines <n>` | Max changed lines per candidate | `30` |
| `--families <list>` | Comma-separated scenario families | all |
| `--dry-run` | Generate candidates but skip validation | `false` |

## Programmatic Usage

```typescript
import { runImproveLoop } from '@sovereign-labs/improve';
import type { TestSurface, ImproveConfig } from '@sovereign-labs/improve';

// Implement the TestSurface interface for your test runner
const surface: TestSurface = {
  packageDir: './my-package',
  selfTestScript: 'scripts/self-test.ts',
  async runBaseline(config) {
    // Run your test suite, return LedgerEntry[]
    return myTestRunner.run(config);
  },
};

const config: ImproveConfig = {
  llm: 'gemini',
  apiKey: process.env.GEMINI_API_KEY,
  maxCandidates: 3,
  maxLines: 30,
  dryRun: false,
};

const entries = await runImproveLoop(surface, config);
const accepted = entries.filter(e => e.verdict === 'accepted');
console.log(`${accepted.length} improvements applied`);
```

### Custom LLM Providers

```typescript
import { createLLMProvider, createClaudeCodeProvider } from '@sovereign-labs/improve';

// Factory — routes to the right provider based on config
const callLLM = createLLMProvider(config);

// Claude Code callback — for interactive Claude Code sessions
const callLLM = createClaudeCodeProvider(async (system, user) => {
  // Your callback sends prompts to Claude Code and returns the response
  return { text: response, inputTokens: 0, outputTokens: 0 };
});
```

### Provider Comparison

| Provider | Best For | Notes |
|----------|----------|-------|
| `gemini` | General use | Gemini 2.5 Flash, fast, cheap |
| `anthropic` | Standard Claude | Claude Sonnet, good reasoning |
| `claude` | Domain-aware | Enhanced prompts with architecture context |
| `claude-code` | Interactive | Callback-based, integrates with Claude Code sessions |
| `ollama` | Air-gap / free | Local models (qwen3:4b default) |
| `none` | Testing | Skips LLM diagnosis and fix generation |

## Key Concepts

### TestSurface Interface

Any test runner can be improved — just implement this interface:

```typescript
interface TestSurface {
  runBaseline(config: TestSurfaceConfig): Promise<LedgerEntry[]>;
  packageDir: string;
  selfTestScript: string;
}
```

### Overfitting Detection

Scenarios are split into three sets:
- **Dirty** — Failing scenarios the fix should repair
- **Validation** — Clean scenarios that must stay clean
- **Holdout** — Withheld clean scenarios for overfitting detection

A fix that passes validation but regresses the holdout is rejected as overfitting.

### Cross-Run Dedup

Fix candidates are SHA-256 hashed. Failed hashes are stored in `data/improve-history.json` and skipped on subsequent runs. The LLM also receives prior attempt context to avoid repeating strategies.

### Bounded Edit Surface

Edits are constrained to a configurable set of files (`DEFAULT_BOUNDED_SURFACE`). Frozen files (`DEFAULT_FROZEN_FILES`) are never modified. This prevents the LLM from rewriting core infrastructure to make tests pass.

### Verdicts

| Verdict | Meaning |
|---------|---------|
| `accepted` | Fix improves dirty scenarios, no regressions, passes holdout |
| `rejected_regression` | Fix causes regressions in clean scenarios |
| `rejected_overfitting` | Fix passes validation but fails holdout |
| `rejected_no_fix` | No valid fix candidates generated |
| `skipped_all_clean` | All scenarios already clean |
| `skipped_no_llm` | Needs LLM but no provider configured |

## Architecture

```
improve/
  src/
    index.ts        — Public API exports
    cli.ts          — CLI entry point
    types.ts        — All type definitions (zero external imports)
    improve.ts      — Main orchestrator (7-step pipeline)
    triage.ts       — Evidence bundling + edit surface guards
    prompts.ts      — LLM prompt construction (generic + Claude-aware)
    providers.ts    — LLM provider factory (Gemini, Anthropic, Claude, Ollama)
    subprocess.ts   — Isolated validation (copy package, apply edits, run tests)
    report.ts       — Terminal output formatting
    utils.ts        — JSON extraction, hashing, retry logic
```

**Zero coupling:** The improve package has zero runtime imports from any Sovereign package. All integration happens at the CLI/harness level via subprocess invocation and the `TestSurface` interface.

## License

MIT