npm package: @frontsail_ai/frontevals
Description: A minimal, vitest-native evals library for LLM applications
Latest version: 0.1.0

# frontevals

A minimal, vitest-native evals library for LLM applications.

## Features

- **Vitest-native**: Works seamlessly inside your existing vitest test suite
- **Simple API**: Just `evalSuite()` or `evalTest()` - no custom CLI needed
- **Built-in metrics**: `exact()`, `contains()`, `startsWith()`, `endsWith()`, `jsonValid()`, `hasKeys()`
- **LLM-as-judge**: G-Eval style custom criteria evaluation via OpenAI
- **Autoevals integration**: Re-exports popular scorers like `Factuality`, `Levenshtein`, etc.
- **Multi-run support**: Run cases multiple times with pass rate thresholds for non-deterministic tasks
- **Pretty output**: Console tables showing scores and pass/fail status

## Installation

```bash
npm install @frontsail_ai/frontevals
```

## Quick Start

### Basic Usage

```typescript
import { describe, it, expect } from 'vitest';
import { evalSuite } from '@frontsail_ai/frontevals';
import { exact, contains } from '@frontsail_ai/frontevals/metrics';

describe('Greeting Evals', () => {
  it('passes all cases', async () => {
    const result = await evalSuite({
      name: 'greet function',
      data: [
        { input: 'Alice', expected: 'Hello, Alice!' },
        { input: 'Bob', expected: 'Hello, Bob!' },
      ],
      task: (name) => `Hello, ${name}!`,
      scorers: [exact(), contains('Hello')],
    });

    expect(result.summary.passRate).toBe(1);
  });
});
```

### Even Simpler: evalTest Helper

```typescript
import { evalTest } from '@frontsail_ai/frontevals/vitest';
import { exact } from '@frontsail_ai/frontevals/metrics';

describe('String Utils', () => {
  evalTest('uppercase works', {
    data: [
      { input: 'hello', expected: 'HELLO' },
      { input: 'world', expected: 'WORLD' },
    ],
    task: (s) => s.toUpperCase(),
    scorers: [exact()],
  });
});
```

## API Reference

### evalSuite(config)

Run an evaluation suite and return structured results.

```typescript
interface EvalSuiteConfig<TInput, TOutput> {
  name: string;                    // Suite name for display
  data: EvalData<TInput>[];        // Test cases (or async function returning them)
  task: (input: TInput) => TOutput | Promise<TOutput>;  // Function to evaluate
  scorers: Scorer<TOutput>[];      // Scoring functions
  runs?: number;                   // Times to run each case (default: 1)
  threshold?: number;              // Pass threshold 0-1 (default: 1.0)
}

interface EvalData<TInput> {
  input: TInput;
  expected?: unknown;
  name?: string;                   // Custom display name
}
```

### evalTest(name, config, threshold?)

Vitest helper that creates a test case automatically.

```typescript
evalTest('test name', {
  data: [...],
  task: (input) => output,
  scorers: [...],
}, 0.9);  // Optional threshold (default: 1.0)
```

## Built-in Metrics

Import from `frontevals/metrics`:

| Metric | Description |
|--------|-------------|
| `exact()` | Exact string match |
| `contains(str)` | Output contains substring |
| `startsWith(str)` | Output starts with prefix |
| `endsWith(str)` | Output ends with suffix |
| `jsonValid()` | Output is valid JSON |
| `hasKeys(keys)` | Output object has required keys |

### Example

```typescript
import { exact, contains, jsonValid, hasKeys } from '@frontsail_ai/frontevals/metrics';

const result = await evalSuite({
  name: 'API Response',
  data: [{ input: 'query', expected: '{"status":"ok"}' }],
  task: async (q) => await api.call(q),
  scorers: [
    jsonValid(),
    hasKeys(['status']),
    contains('ok'),
  ],
});
```

## G-Eval (LLM-as-Judge)

Use custom criteria evaluated by an LLM:

```typescript
import { gEval } from '@frontsail_ai/frontevals/metrics';

const result = await evalSuite({
  name: 'Support Bot',
  data: [
    { input: 'My order is late', expected: 'Apologize and offer help' },
  ],
  task: async (q) => await bot.respond(q),
  scorers: [
    gEval({ criteria: 'Is the response empathetic and helpful?' }),
    gEval({
      name: 'actionable',
      criteria: 'Does the response provide a clear next step?',
      threshold: 0.8,
    }),
  ],
});
```

### gEval Options

```typescript
gEval({
  name?: string;           // Scorer name (default: 'gEval')
  criteria: string;        // Plain language criteria
  steps?: string[];        // Optional evaluation steps
  threshold?: number;      // Pass threshold (default: 0.7)
  model?: string;          // OpenAI model (default: 'gpt-4o-mini')
})
```

**Note:** Requires `OPENAI_API_KEY` environment variable.

## Autoevals Integration

Re-exported scorers from the [autoevals](https://github.com/braintrustdata/autoevals) library:

```typescript
import { Factuality, Levenshtein } from '@frontsail_ai/frontevals/autoevals';

const result = await evalSuite({
  name: 'Chatbot Quality',
  data: [
    { input: 'What is TypeScript?', expected: 'A typed superset of JavaScript' },
  ],
  task: async (q) => await chatbot.answer(q),
  scorers: [Factuality, Levenshtein],
});
```

Available: `Factuality`, `Levenshtein`, `ClosedQA`, `Battle`, `Humor`, `Security`, `Summary`, `Translation`

## Multiple Runs (Non-Deterministic Tasks)

For LLM tasks with variable outputs, run multiple times and set a pass threshold:

```typescript
const result = await evalSuite({
  name: 'Chatbot Consistency',
  data: [
    { input: 'Explain recursion', expected: 'A clear explanation' },
  ],
  task: async (q) => await chatbot.answer(q),
  scorers: [gEval({ criteria: 'Is the explanation clear?' })],
  runs: 5,          // Run each case 5 times
  threshold: 0.8,   // 80% of runs must pass
});

console.log(result.results[0].passRate); // e.g., 0.8
```

## Console Output

### Single Run Mode

```
┌─────────────────────────────────────────────────────────────────┐
│                     greet function                              │
├──────────┬────────────────┬───────┬──────────┬─────────┬────────┤
│ Case     │ Output         │ exact │ contains │ Score   │ Status │
├──────────┼────────────────┼───────┼──────────┼─────────┼────────┤
│ Alice    │ Hello, Alice!  │ 1.00  │ 1.00     │ 1.00    │ ✓ PASS │
│ Bob      │ Hello, Bob!    │ 1.00  │ 1.00     │ 1.00    │ ✓ PASS │
├──────────┴────────────────┴───────┴──────────┴─────────┴────────┤
│ Summary: 2/2 passed (100%)                                      │
└─────────────────────────────────────────────────────────────────┘
```

### Multiple Runs Mode

```
┌───────────────────────────────────────────────────────────────────────┐
│                     Chatbot Consistency (5 runs, 80% threshold)       │
├────────────────────┬─────────┬───────────┬──────────┬─────────────────┤
│ Case               │ gEval   │ Runs      │ PassRate │ Status          │
├────────────────────┼─────────┼───────────┼──────────┼─────────────────┤
│ Explain recursion  │ 0.85    │ 4/5       │ 80%      │ ✓ PASS          │
│ What is a closure  │ 0.60    │ 3/5       │ 60%      │ ✗ FAIL (< 80%)  │
├────────────────────┴─────────┴───────────┴──────────┴─────────────────┤
│ Summary: 1/2 cases passed | 7/10 total trials passed (70%)            │
└───────────────────────────────────────────────────────────────────────┘
```

## Result Types

```typescript
interface SuiteResult {
  name: string;
  results: EvalResult[];
  summary: {
    total: number;
    passed: number;
    failed: number;
    passRate: number;
    totalTrials: number;
    byScorer: Record<string, { avgScore: number; passRate: number }>;
  };
  runs: number;
  threshold: number;
}

interface EvalResult {
  name: string;
  input: unknown;
  expected?: unknown;
  trials: TrialResult[];
  passRate: number;
  pass: boolean;
}

interface TrialResult {
  trialIndex: number;
  output: unknown;
  scores: Array<{ name: string; score: number; pass: boolean }>;
  pass: boolean;
}
```

## Custom Scorers

Create custom scorers matching the autoevals signature:

```typescript
import type { Scorer } from '@frontsail_ai/frontevals';

const lengthScorer: Scorer<string> = ({ output, expected }) => {
  const diff = Math.abs(output.length - String(expected).length);
  const score = Math.max(0, 1 - diff / 100);
  return { score, name: 'length' };
};

// Async scorers are supported
const apiScorer: Scorer<string> = async ({ output }) => {
  const result = await externalApi.evaluate(output);
  return { score: result.score, name: 'api', metadata: result };
};
```

---

## Development

### Setup

```bash
git clone <repo-url>
cd frontevals
npm install
```

### Scripts

```bash
npm test          # Run tests
npm run test:watch # Run tests in watch mode
npm run build     # Build TypeScript
```

### Running Tests with Coverage

```bash
npx vitest run --coverage
```

### Project Structure

```
frontevals/
├── src/
│   ├── index.ts          # Main exports
│   ├── types.ts          # TypeScript interfaces
│   ├── eval-suite.ts     # Core evalSuite function
│   ├── reporter.ts       # Console table output
│   ├── vitest.ts         # evalTest helper
│   ├── autoevals.ts      # Re-exports from autoevals
│   └── metrics/
│       ├── index.ts      # Built-in metrics
│       └── geval.ts      # G-Eval LLM-as-judge
├── tests/
│   ├── eval-suite.test.ts
│   ├── metrics.test.ts
│   ├── geval.test.ts
│   ├── reporter.test.ts
│   ├── vitest-helper.test.ts
│   ├── exports.test.ts
│   └── output.test.ts
├── package.json
├── tsconfig.json
└── vitest.config.ts
```

### Adding New Metrics

1. Create scorer function in `src/metrics/index.ts` or a new file
2. Export from `src/metrics/index.ts`
3. Add tests in `tests/metrics.test.ts`

Example:

```typescript
// src/metrics/index.ts
export function regex(pattern: RegExp): Scorer<unknown> {
  return ({ output }): ScoreResult => {
    const score = pattern.test(String(output)) ? 1 : 0;
    return { score, name: 'regex' };
  };
}
```

### Dependencies

- **autoevals**: LLM evaluation metrics
- **console-table-printer**: Pretty console tables
- **openai**: Required for gEval

### Peer Dependencies

- **vitest**: >=1.0.0

## License

MIT