npm package: @scoutos/evals
Description: Scout Evaluation SDK - Evaluate AI agent outputs
Author: Scout
Latest version: 0.1.0

# @scoutos/evals

Scout Evaluation SDK for TypeScript - Evaluate AI agent outputs with type-safe assertions.

## Table of Contents

- [What is this?](#what-is-this)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [Client](#client)
  - [Creating a Client](#creating-a-client)
  - [Choosing a Method](#choosing-a-method)
  - [evaluate()](#evaluate)
  - [run()](#run)
  - [runDataset()](#rundataset)
- [Check Builders](#check-builders)
  - [Field Checks](#field-checks)
  - [Agent Checks](#agent-checks)
  - [Custom Checks](#custom-checks)
  - [Scorer References](#scorer-references)
- [Config Files](#config-files)
  - [YAML Format](#yaml-format)
  - [JSON Format](#json-format)
  - [Loading Config Files](#loading-config-files)
- [Evaluation Results](#evaluation-results)
- [Error Handling](#error-handling)
- [CI Integration](#ci-integration)
- [TypeScript Types](#typescript-types)
- [Troubleshooting](#troubleshooting)

---

## What is this?

`@scoutos/evals` is an SDK for evaluating AI agent outputs. Use it to:

- **Assert on outputs** — Check fields, types, patterns, and values
- **Use AI as a judge** — Let Claude evaluate subjective qualities like helpfulness and accuracy
- **Run in CI** — Fail builds when quality drops below thresholds
- **Track over time** — Named evaluations group runs so you can spot regressions

Results are stored in the [Scout dashboard](https://app.scoutos.com) where you can view history, compare runs, and debug failures.

**Get your API key:** [app.scoutos.com/settings/api-keys](https://app.scoutos.com/settings/api-keys)

---

## Installation

```bash
npm install @scoutos/evals
```

## Quick Start

```typescript
import { createClient, checks } from "@scoutos/evals";

const client = createClient(); // Uses SCOUT_API_KEY env var

const result = await client.evaluate({
  name: "my-agent-tests",
  tests: [
    {
      input: { query: "hello" },
      output: { response: "Hi there!" },
    },
  ],
  assert: [
    checks.field("response").exists(),
    checks.agent("Is this a friendly greeting?"),
  ],
});

console.log(result.passed);   // true
console.log(result.passRate); // 1.0
console.log(result.runUrl);   // https://app.scoutos.com/evaluations/runs/...
```

---

## Client

### Creating a Client

```typescript
import { createClient } from "@scoutos/evals";

// Uses SCOUT_API_KEY environment variable
const client = createClient();

// Or pass API key directly
const client = createClient("sk_...");

// Or full configuration
const client = createClient({
  apiKey: "sk_...",
  baseUrl: "https://api.scoutos.com", // Optional
});
```

### Choosing a Method

| Method | Use When |
|--------|----------|
| `evaluate()` | You already have outputs (from logs, a database, or another system) |
| `run()` | You want to execute a function and evaluate its outputs in one step |
| `runDataset()` | You have a dataset stored in Scout and want to run your function against it |

**Quick decision:**
- Have outputs already? → `evaluate()`
- Want to run code and evaluate? → `run()`
- Using Scout datasets? → `runDataset()`

### evaluate()

Evaluate pre-computed outputs against assertions.

```typescript
const result = await client.evaluate({
  // Name for tracking across runs (recommended)
  name: "product-search-tests",

  // Metadata for tracking versions
  metadata: {
    version: "v2.1",
    model: "gpt-4",
    gitCommit: "abc123",
  },

  // Test cases
  tests: [
    {
      name: "basic-search",
      input: { query: "laptops" },
      output: { results: [{ name: "MacBook", price: 1299 }] },
      assert: [checks.field("results").length.gte(1)], // Per-test assertions
    },
  ],

  // Global assertions (applied to all tests)
  assert: [
    checks.field("results").exists(),
    checks.agent("Are these results relevant to the query?"),
  ],

  // Tags for filtering in Scout UI
  tags: ["v1.0", "production"],
});
```

**Single test shorthand:**

```typescript
const result = await client.evaluate({
  name: "single-test",
  input: { query: "hello" },
  output: { response: "Hi!" },
  assert: [checks.field("response").exists()],
});
```

### run()

Execute a function against inputs and evaluate the outputs.

```typescript
const result = await client.run(
  // Inputs
  [
    { query: "laptops" },
    { query: "phones" },
  ],

  // Function to execute (sync or async)
  async (input) => {
    return await mySearchFunction(input.query);
  },

  // Options
  {
    name: "search-function-tests",
    metadata: { version: "v1.0" },
    assert: [
      checks.field("results").exists(),
      checks.field("results").length.gte(1),
    ],
    tags: ["integration"],
  }
);
```

The SDK automatically measures latency for each execution.

### runDataset()

Run a function against a dataset stored in Scout.

```typescript
const result = await client.runDataset({
  dataset: "product-search-golden-v2", // Dataset name or ID from Scout UI
  fn: async (input) => {
    return await mySearchFunction(input.query);
  },
  name: "dataset-regression-test",
  assert: [
    checks.field("results").exists(),
    checks.agent("Are results accurate?"),
  ],
});
```

---

## Check Builders

### Field Checks

Assert on any field in the output using a fluent builder.

```typescript
import { checks } from "@scoutos/evals";

// Existence
checks.field("results").exists()

// Equality
checks.field("status").equals("success")
checks.field("status").notEquals("error")

// Numeric comparisons
checks.field("count").gt(0)       // greater than
checks.field("count").gte(1)      // greater than or equal
checks.field("count").lt(100)     // less than
checks.field("count").lte(99)     // less than or equal

// String/array contains
checks.field("message").includes("hello")   // substring
checks.field("tags").includes("urgent")     // array element

// Regex matching
checks.field("email").matches(/^.+@.+\..+$/)

// Type checking
checks.field("count").isType("number")
checks.field("items").isType("array")
checks.field("data").isType("object")

// Length (for strings and arrays)
checks.field("items").length.gte(5)
checks.field("name").length.lte(100)
```

**Nested paths:**

```typescript
// Dot notation
checks.field("user.profile.name").exists()

// Array indices
checks.field("items[0].name").equals("first")

// Mixed
checks.field("users[0].profile.email").matches(/^.+@.+$/)
```

### Agent Checks

Use AI to evaluate subjective qualities.

```typescript
// Simple prompt
checks.agent("Is this response helpful and accurate?")

// With criteria and weights
checks.agent({
  prompt: "Evaluate the quality of this search result",
  criteria: [
    {
      name: "relevance",
      description: "Are results relevant to the query?",
      weight: 2.0,
    },
    {
      name: "completeness",
      description: "Does the response cover all aspects?",
      weight: 1.0,
    },
  ],
  model: "claude-sonnet-4", // Default model
})
```

**Available models:**
- `claude-sonnet-4` (default) — Good balance of speed and quality
- `claude-opus-4` — Highest quality, slower and more expensive
- `claude-haiku` — Fastest, best for simple checks

### Custom Checks

Define assertions that run locally in your environment.

```typescript
// Return boolean
checks.custom("has-items", (input, output) => {
  return output.items && output.items.length > 0;
})

// Return detailed result
checks.custom("price-valid", (input, output) => ({
  passed: output.price > 0 && output.price < 10000,
  score: output.price > 0 ? 1.0 : 0.0,
  reason: output.price <= 0 ? "Price must be positive" : undefined,
  metadata: { actualPrice: output.price },
}))

// Async check (call external APIs, databases, etc.)
checks.custom("api-validation", async (input, output) => {
  const isValid = await validateWithExternalAPI(output);
  return { passed: isValid };
})
```

### Scorer References

Reference scorers saved in Scout UI.

```typescript
checks.scorer("scorer_abc123")
```

---

## Config Files

Define evaluations in YAML or JSON for version control and sharing.

### YAML Format

```yaml
# eval.yaml
name: product-search-tests

metadata:
  version: "1.0"
  environment: staging

tests:
  - name: laptop-search
    input:
      query: "gaming laptops"
    output:
      results:
        - name: "ASUS ROG"
          price: 1299
    assert:
      - field: results
        exists: true
      - field: results
        length:
          gte: 1
      - agent: "Are these gaming laptops?"

# Global assertions
assert:
  - field: results
    exists: true

tags:
  - regression
```

### JSON Format

```json
{
  "name": "product-search-tests",
  "tests": [
    {
      "name": "basic-search",
      "input": { "query": "laptops" },
      "output": { "results": [{ "name": "MacBook" }] },
      "assert": [
        { "field": "results", "exists": true },
        { "agent": "Are results relevant?" }
      ]
    }
  ]
}
```

### Loading Config Files

```typescript
import { loadConfig, createClient } from "@scoutos/evals";

const config = await loadConfig("./eval.yaml");
const client = createClient();
const result = await client.evaluate(config);
```

---

## Evaluation Results

```typescript
const result = await client.evaluate({ ... });

// Overall status
result.passed      // boolean — did all tests pass?
result.passRate    // number — 0.0 to 1.0
result.passCount   // number
result.failCount   // number
result.totalCount  // number

// Performance
result.avgLatencyMs // number — average latency in ms

// Details
result.results     // TestResult[] — all test results
result.failures    // TestResult[] — only failed tests

// Scout UI
result.runId       // string
result.runUrl      // string — link to view in dashboard

// Get specific test by name
const test = result.getTest("laptop-search");

// Full API response
result.raw
```

---

## Error Handling

```typescript
import { createClient, checks } from "@scoutos/evals";

const client = createClient();

try {
  const result = await client.evaluate({ ... });
} catch (error) {
  if (error.message.includes("401")) {
    // Invalid or expired API key
    console.error("Authentication failed. Check your SCOUT_API_KEY.");
  } else if (error.message.includes("429")) {
    // Rate limited
    console.error("Rate limited. Slow down requests or contact support.");
  } else if (error.message.includes("5")) {
    // Server error (500, 502, 503, etc.)
    console.error("Scout API is temporarily unavailable. Retry later.");
  } else {
    // Network error or other issue
    console.error("Evaluation failed:", error.message);
  }
}
```

**Common errors:**

| Status | Meaning | Solution |
|--------|---------|----------|
| 401 | Invalid API key | Check `SCOUT_API_KEY` is set correctly |
| 403 | Forbidden | Your API key doesn't have access to this resource |
| 404 | Not found | Dataset or scorer ID doesn't exist |
| 429 | Rate limited | Add delays between requests or contact support |
| 500+ | Server error | Retry with exponential backoff |

---

## CI Integration

```typescript
import { createClient, checks } from "@scoutos/evals";

async function runEvaluation() {
  const client = createClient();

  const result = await client.evaluate({
    name: "ci-regression-tests",
    metadata: {
      gitCommit: process.env.GIT_COMMIT,
      branch: process.env.GIT_BRANCH,
    },
    tests: [...],
    assert: [...],
  });

  console.log(`Pass rate: ${(result.passRate * 100).toFixed(1)}%`);
  console.log(`Results: ${result.passCount}/${result.totalCount}`);
  console.log(`Details: ${result.runUrl}`);

  if (!result.passed) {
    console.error("\nFailed tests:");
    for (const failure of result.failures) {
      console.error(`  ${failure.test_name}`);
      for (const scorer of failure.scorer_results.filter(s => !s.passed)) {
        console.error(`    - ${scorer.scorer_id}: ${scorer.reason}`);
      }
    }
    process.exit(1);
  }
}

runEvaluation();
```

**GitHub Actions:**

```yaml
name: AI Evaluation

on: [push, pull_request]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
      - run: npm ci
      - name: Run Evaluations
        env:
          SCOUT_API_KEY: ${{ secrets.SCOUT_API_KEY }}
          GIT_COMMIT: ${{ github.sha }}
          GIT_BRANCH: ${{ github.ref_name }}
        run: npx tsx scripts/evaluate.ts
```

---

## TypeScript Types

Key interfaces for IDE autocompletion and type safety:

```typescript
interface TestCase {
  name?: string;
  input?: Record<string, unknown>;
  output: Record<string, unknown>;
  expected_output?: Record<string, unknown>; // For custom comparison checks
  assert?: Check[];
  latency_ms?: number;
}

interface TestResult {
  test_name: string;
  passed: boolean;
  status: "pass" | "fail" | "error";
  input: Record<string, unknown>;
  output: Record<string, unknown>;
  latency_ms: number;
  scorer_results: ScorerResult[];
  composite_score: number;
}

interface ScorerResult {
  scorer_id: string;
  passed: boolean;
  score: number;          // 0.0 to 1.0
  reason?: string;        // Why it failed
  metadata?: Record<string, unknown>;
}

interface EvaluationMetrics {
  pass_rate: number;
  pass_count: number;
  fail_count: number;
  total_count: number;
  avg_latency_ms: number;
}

// Check is a union of all check types
type Check =
  | FieldExistsCheck
  | ExactMatchCheck
  | ContainsCheck
  | RegexMatchCheck
  | TypeCheckCheck
  | AgentCheck
  | CustomCheck;
```

---

## Troubleshooting

**"API key required" error**

Make sure `SCOUT_API_KEY` is set:
```bash
export SCOUT_API_KEY=sk_...
```

Or pass it directly:
```typescript
const client = createClient("sk_...");
```

**Tests pass locally but fail in CI**

- Check that `SCOUT_API_KEY` is set in your CI secrets
- Ensure the secret name matches exactly (case-sensitive)

**Agent checks are slow**

Agent checks use AI models which have latency. Options:
- Use `claude-haiku` for faster checks: `checks.agent({ prompt: "...", model: "claude-haiku" })`
- Run fewer agent checks, more field checks
- Run evaluations in parallel if you have multiple test suites

**Custom check isn't running**

Custom checks run locally, not on Scout servers. Make sure:
- The function doesn't throw unhandled errors
- You're returning `boolean` or `{ passed: boolean, ... }`

**Rate limiting**

If you're running many evaluations in a loop, add delays:
```typescript
for (const test of tests) {
  await client.evaluate(test);
  await new Promise(r => setTimeout(r, 100)); // 100ms delay
}
```

---

## License

MIT