npm package: @safetnsr/autotune
Description: GPU-poor autoresearch — calibrate any scoring tool against ground truth
Author: safetnsr
Latest version: 0.1.0

# autotune

GPU-poor autoresearch. calibrate any scoring tool by iterating parameters against ground truth.

inspired by [karpathy/autoresearch](https://github.com/karpathy/autoresearch) — the same pattern (iterate, measure, keep or discard) applied without a GPU.

## what it does

you have a scoring tool with tunable parameters. you have a dataset with expected results. autotune:

1. mutates your parameters
2. runs your evaluation
3. measures the metric
4. keeps improvements, discards failures
5. repeats

no machine learning. no GPU. just systematic parameter search with a clear metric.

## quickstart

```bash
npx @safetnsr/autotune init
```

this creates three files:

- **`autotune.json`** — defines your parameters and their ranges
- **`params.json`** — your current parameter values
- **`eval.js`** — your evaluation script (you implement this)

edit `eval.js` to run your tool against a dataset and output a metric. then:

```bash
npx @safetnsr/autotune run --iterations 100
```

autotune will iterate 100 times, logging each attempt and keeping only improvements.

## example: calibrating a code quality scorer

we used autotune to calibrate [vet](https://github.com/safetnsr/vet) against 43 public repos. correlation went from -0.32 to +0.83 across 13 iterations.

```json
{
  "name": "vet-calibration",
  "params": "thresholds.json",
  "eval": "bash run-calibration.sh",
  "metric": "correlation",
  "higherIsBetter": true,
  "maxIterations": 50,
  "strategy": "hill-climb",
  "paramDefs": [
    {
      "path": "category_weights",
      "type": "weight-group",
      "step": 0.05,
      "members": [
        "category_weights.security",
        "category_weights.integrity",
        "category_weights.debt",
        "category_weights.deps"
      ]
    },
    {
      "path": "integrity.empty_catch_error_penalty",
      "type": "number",
      "min": 1,
      "max": 15,
      "step": 1
    }
  ]
}
```

full writeup: [from -0.32 to +0.83](https://comrade.md/writing/calibrating-vet-with-autoresearch)

## use cases

- **scoring algorithms** — calibrate weights and thresholds against labeled data
- **prompt optimization** — iterate on prompt templates, measure output quality
- **config tuning** — optimize build configs, deploy settings, rate limits
- **any parameter search** — if you can measure it, you can autotune it

## how it works

autotune uses hill-climbing with random restarts:

1. pick 1-3 random parameters
2. mutate each by 1-3 steps in a random direction
3. run eval, extract metric
4. if metric improved → keep new params
5. if metric worsened → revert to previous best
6. repeat

results are logged as JSONL in `.autotune/` — every iteration, every change, every metric.

## cli

```bash
autotune init                     # scaffold project
autotune run [--iterations N]     # run optimization loop
autotune status                   # show latest run summary
autotune help                     # show help
```

## programmatic

```typescript
import { run } from '@safetnsr/autotune';

const config = JSON.parse(fs.readFileSync('autotune.json', 'utf-8'));
const summary = run(config, process.cwd());

console.log(`improved from ${summary.startMetric} to ${summary.bestMetric}`);
```

## license

MIT