npm package: @manet/agent-snapshot
Description: A snapshot-based agent test framework for `@manet/agent` based Agents
Latest version: 0.0.4

# @manet/agent-snapshot

A snapshot-based agent test framework for `@manet/agent` based Agents.

## Overview

`@manet/agent-snapshot` provides a comprehensive testing framework for AI agents built with `@manet/agent`. It enables you to:

- **Generate snapshots** of real agent interactions with LLMs
- **Replay tests** using captured snapshots for deterministic, fast testing
- **Verify behavior** by comparing expected vs actual responses, tool calls, and event streams
- **Normalize dynamic data** to ignore timestamps, IDs, and other volatile values

## Installation

```bash
npm install @manet/agent-snapshot
```

## Quick Start

### Basic Usage

```typescript
import { AgentSnapshot } from '@manet/agent-snapshot';
import { Agent } from '@manet/agent';

// Create your agent instance
const agent = new Agent({
  // ... your agent configuration
});

// Create snapshot instance
const agentSnapshot = new AgentSnapshot(agent, {
  snapshotPath: './fixtures/my-test-case',
  snapshotName: 'my-test-case'
});

// Generate snapshots from real LLM interactions
await agentSnapshot.generate({
  input: 'Hello, how can you help me?'
});

// Run tests using captured snapshots
const result = await agentSnapshot.replay({
  input: 'Hello, how can you help me?'
});
```

### Using as a Transparent Wrapper

```typescript
// AgentSnapshot can be used as a drop-in replacement for Agent
const response = await agentSnapshot.run({
  input: 'What is the weather today?',
  stream: false
});

// Or with streaming
for await (const event of await agentSnapshot.run({
  input: 'Tell me a story',
  stream: true
})) {
  console.log(event);
}
```

## Core Concepts

### Snapshots

Snapshots capture the complete state of agent interactions:

- **LLM Requests**: Prompts sent to the language model
- **LLM Responses**: Model responses (including streaming chunks)
- **Tool Calls**: Function calls executed by the agent
- **Event Streams**: Complete agent event timeline

### Directory Structure

```
fixtures/
└── my-test-case/
    ├── initial/
    │   └── event-stream.jsonl
    ├── loop-1/
    │   ├── llm-request.jsonl
    │   ├── llm-response.jsonl
    │   ├── tool-calls.jsonl
    │   └── event-stream.jsonl
    ├── loop-2/
    │   └── ...
    └── event-stream.jsonl
```

## API Reference

### AgentSnapshot

The main class for managing agent snapshots and testing.

#### Constructor

```typescript
new AgentSnapshot(agent: Agent, options: AgentSnapshotOptions)
```

**Options:**
- `snapshotPath: string` - Directory to store snapshots
- `snapshotName?: string` - Name for the snapshot
- `updateSnapshots?: boolean` - Whether to update existing snapshots
- `normalizerConfig?: AgentNormalizerConfig` - Configuration for data normalization
- `verification?: VerificationOptions` - What to verify during tests

#### Methods

##### `generate(runOptions)`

Generate snapshots by executing the agent with real LLM calls.

```typescript
const result = await agentSnapshot.generate({
  input: 'Your test input here'
});
```

**Returns:** `SnapshotGenerationResult`

##### `replay(runOptions, config?)`

Run tests using previously captured snapshots.

```typescript
const result = await agentSnapshot.replay({
  input: 'Your test input here'
}, {
  updateSnapshots: false,
  verification: {
    verifyLLMRequests: true,
    verifyEventStreams: true,
    verifyToolCalls: true
  }
});
```

**Returns:** `SnapshotRunResult`

##### `run(runOptions)`

Transparent wrapper around the agent's run method that generates snapshots.

```typescript
// Non-streaming
const response = await agentSnapshot.run({
  input: 'Your input',
  stream: false
});

// Streaming
for await (const event of await agentSnapshot.run({
  input: 'Your input',
  stream: true
})) {
  console.log(event);
}
```

### Configuration

#### AgentNormalizerConfig

Configure how dynamic data is normalized during comparisons.

```typescript
const normalizerConfig: AgentNormalizerConfig = {
  fieldsToNormalize: [
    { pattern: /id$/, replacement: '<<ID>>' },
    { pattern: 'timestamp', replacement: '<<TIMESTAMP>>' },
    { pattern: /Time$/, replacement: '<<TIMESTAMP>>' }
  ],
  fieldsToIgnore: [
    'debugInfo',
    /internal_/
  ],
  customNormalizers: [
    {
      pattern: 'customField',
      normalizer: (value, path) => normalizeCustomValue(value)
    }
  ]
};
```

#### VerificationOptions

Control what aspects of agent behavior are verified.

```typescript
const verification: VerificationOptions = {
  verifyLLMRequests: true,    // Verify LLM prompts match
  verifyEventStreams: true,   // Verify event streams match
  verifyToolCalls: true       // Verify tool calls match
};
```

## Advanced Usage

### Custom Normalization

Handle dynamic data that shouldn't affect test results:

```typescript
const agentSnapshot = new AgentSnapshot(agent, {
  snapshotPath: './fixtures/test',
  normalizerConfig: {
    fieldsToNormalize: [
      { pattern: 'sessionId', replacement: '<<SESSION_ID>>' },
      { pattern: 'elapsedMs', replacement: '<<ELAPSED_MS>>' },
      { pattern: /_at$/, replacement: '<<TIMESTAMP>>' }
    ],
    customNormalizers: [
      {
        pattern: 'userMessage',
        normalizer: (value, path) => {
          // Normalize user IDs in messages
          if (typeof value === 'string') {
            return value.replace(/user-\d+/g, 'user-<<ID>>');
          }
          return value;
        }
      }
    ]
  }
});
```

### Selective Verification

Choose what to verify based on your test needs:

```typescript
// Only verify LLM requests, ignore timing differences
const result = await agentSnapshot.replay(runOptions, {
  verification: {
    verifyLLMRequests: true,
    verifyEventStreams: false,  // Skip event stream verification
    verifyToolCalls: false      // Skip tool call verification
  }
});
```

### Updating Snapshots

Update snapshots when behavior changes intentionally:

```typescript
// Update mode - skips verification, updates snapshots
const result = await agentSnapshot.replay(runOptions, {
  updateSnapshots: true
});
```

### Batch Testing with AgentSnapshotRunner

For testing multiple scenarios:

```typescript
import { AgentSnapshotRunner } from '@manet/agent-snapshot';

const runner = new AgentSnapshotRunner([
  {
    name: 'basic-conversation',
    path: './test-cases/basic-conversation.js',
    snapshotPath: './fixtures/basic-conversation',
    vitestSnapshotPath: './tests/snapshots/basic-conversation'
  },
  {
    name: 'tool-usage',
    path: './test-cases/tool-usage.js',
    snapshotPath: './fixtures/tool-usage',
    vitestSnapshotPath: './tests/snapshots/tool-usage'
  }
]);

// Generate all snapshots
await runner.generateAll();

// Run all tests
await runner.replayAll();
```

## Testing Framework Integration

### Vitest Integration

```typescript
import { test, expect } from 'vitest';
import { AgentSnapshot } from '@manet/agent-snapshot';

test('agent handles basic conversation', async () => {
  const agent = createTestAgent();
  const snapshot = new AgentSnapshot(agent, {
    snapshotPath: './fixtures/basic-conversation'
  });

  const result = await snapshot.replay({
    input: 'Hello, how are you?'
  });

  expect(result.events).toHaveLength(expectedEventCount);
  expect(result.meta.loopCount).toBe(expectedLoopCount);
});
```

### CI/CD Integration

```bash
# Generate snapshots (only when needed)
npm run test:snapshots:generate

# Run tests against snapshots
npm run test:snapshots

# Update snapshots (when behavior changes)
npm run test:snapshots:update
```

## Best Practices

### 1. Organize Tests Logically

```
fixtures/
├── user-interactions/
│   ├── basic-chat/
│   ├── complex-queries/
│   └── error-handling/
├── tool-usage/
│   ├── single-tools/
│   ├── multi-tools/
│   └── tool-errors/
└── edge-cases/
    ├── time-outs/
    ├── rate-limits/
    └── network-errors/
```

### 2. Use Descriptive Names

```typescript
// Good
const snapshot = new AgentSnapshot(agent, {
  snapshotPath: './fixtures/user-interactions/basic-chat/greeting-response'
});

// Avoid
const snapshot = new AgentSnapshot(agent, {
  snapshotPath: './fixtures/test1'
});
```

### 3. Normalize Volatile Data

Always normalize timestamps, IDs, and other dynamic values:

```typescript
const normalizerConfig = {
  fieldsToNormalize: [
    { pattern: /id$/, replacement: '<<ID>>' },
    { pattern: 'timestamp', replacement: '<<TIMESTAMP>>' },
    { pattern: 'requestId', replacement: '<<REQUEST_ID>>' }
  ]
};
```

### 4. Version Control Snapshots

Commit snapshots to version control to track behavior changes:

```bash
# Add snapshots to git
git add fixtures/
git commit -m "feat: add snapshots for user greeting flow"

# Update snapshots when behavior changes
npm run test:snapshots:update
git add fixtures/
git commit -m "feat: update snapshots for improved response format"
```

## Troubleshooting

### Common Issues

#### "No snapshot found" Error

Ensure snapshots are generated before running tests:

```bash
# Generate snapshots first
await agentSnapshot.generate(runOptions);

# Then run tests
await agentSnapshot.replay(runOptions);
```

#### "Loop count mismatch" Error

This indicates the agent behaved differently during test vs snapshot generation:

```typescript
// Check loop counts in generation result
const genResult = await agentSnapshot.generate(runOptions);
console.log('Generated loops:', genResult.loopCount);

// Compare with test result
const testResult = await agentSnapshot.replay(runOptions);
console.log('Test loops:', testResult.meta.loopCount);
```

#### "Event stream doesn't match" Error

Use normalizer to ignore expected differences:

```typescript
const snapshot = new AgentSnapshot(agent, {
  snapshotPath: './fixtures/test',
  normalizerConfig: {
    fieldsToIgnore: ['timing', 'debugInfo'],
    fieldsToNormalize: [
      { pattern: /elapsedMs$/, replacement: '<<ELAPSED_MS>>' }
    ]
  }
});
```

### Debug Mode

Enable detailed logging:

```typescript
import { logger } from '@manet/agent-snapshot';

// Enable debug logging
logger.setLevel('debug');
```

## Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit a pull request

## License

Apache-2.0 © Bytedance, Inc.