npm package: @sanity/agent-context-explorer
Description: Exploration tool for Sanity Agent Context — produces knowledge docs for production AI agents
Author: Sanity.io
Latest version: 0.0.4

# Agent Context Explorer

Companion tool for [Sanity Agent Context](https://github.com/sanity-io/agent-context). Explores your Agent Context server, documents what works and what doesn't, and produces `exploration-results.md` — ready to copy directly into your Agent Context Document.

## Why This Tool?

When building AI agents that work with Agent Context, agents need to know:
- What data exists and what's reliably populated
- How to query effectively for different question types
- What caveats and hazards will trip them up (the most valuable knowledge)

Manual exploration is time-consuming and doesn't scale. This tool automates that process, producing structured documentation that helps production agents work correctly from day one.

## What You Get

After running the explorer, you'll have documentation that tells agents:
- Which document types to query (and which to ignore)
- Working query patterns for different question types
- **Critical hazards** — data gaps, external system dependencies, and silent failure modes that would otherwise cause wrong answers

## Installation

```bash
# Global install (recommended — reuse across projects)
npm install -g @sanity/agent-context-explorer

# Or install per-project
npm install @sanity/agent-context-explorer
```

## Quick Start

1. Create a questions file (`questions.json`). Always include `expected_answer` — the explorer uses it to guide exploration and validate results:

```json
{
  "questions": [
    { "question": "What sizes does the Trailblazer Hiking Boot come in?", "expected_answer": "US 7-13, including half sizes" },
    { "question": "Is the Ultralight Tent waterproof?", "expected_answer": "Yes, it has a 3000mm waterproof rating with taped seams" },
    { "question": "What's the difference between the Summit Pack and the Daybreak Pack?", "expected_answer": "Summit is 65L for multi-day trips with a frame; Daybreak is 28L for day hikes" }
  ]
}
```

2. Run the explorer:

```bash
agent-context-explorer \
  --mcp-url https://api.sanity.io/vX/agent-context/YOUR_PROJECT_ID/YOUR_DATASET/YOUR_SLUG \
  --questions ./questions.json \
  --sanity-token $SANITY_API_READ_TOKEN \
  --anthropic-api-key $ANTHROPIC_API_KEY
```

3. Copy the contents of `exploration-results.md` into your Agent Context Document's `instructions` field. The output directory will be timestamped (e.g., `./explorer-output-2026-02-11T09-22-30/`).

## Question File Format

```typescript
{
  "questions": [
    {
      "question": string,          // The question to explore (required)
      "expected_answer"?: string,  // Strongly recommended — guides exploration and validates results
      "id"?: string                // Auto-generated as q1, q2, ... if omitted
    }
  ]
}
```

**Always include `expected_answer`.** The explorer uses it in two ways:
1. **Guides exploration** — injected as a hint so the agent knows what to look for
2. **Validates results** — the agent's answer is graded against yours, and this flows into confidence levels in the output

Without expected answers, the explorer can't tell if it found the right data. Even a rough or partial answer is better than none.

**Tips for writing good questions:**
- **Always provide `expected_answer`** — even approximate answers help the explorer validate findings
- Cover different question types (specs, compatibility, comparisons, troubleshooting)
- Include questions you expect to succeed AND ones that might fail
- 20-50 questions is typically sufficient for good coverage. More questions = richer documentation
- **Use an AI assistant to help generate questions** — start by listing questions your customers actually ask, then use an AI to expand them into a diverse set covering different categories and edge cases

## Output Files

The tool generates several files in the output directory:

### `exploration-results.md` (Primary Output)

**This is the file you copy into your Agent Context Document.** It contains LLM-ready instructions with five sections:

1. **Schema Reference** — Document types, what they're used for, and key fields
2. **Query Patterns** — Working query examples organized by use case, with inline warnings
3. **Critical Rules** — "Always do X" and "Never do Y" statements derived from exploration
4. **Known Limitations** — What data is NOT available (null fields, external dependencies)
5. **Exploration Coverage** — What was tested and confidence levels

The synthesis agent may add additional sections when findings naturally cluster (e.g., "Locale Handling" if locale issues were prominent).

**Failures are the most valuable output** — they document what would trip up a naive agent, preventing wrong answers and wasted queries.

### `logs/*.json`

Individual exploration logs for each question, containing:
- All tool calls made
- Learnings and caveats discovered
- Validation results against expected answers

### `metrics.json`

Aggregated statistics: success rates, confidence distribution, category coverage, and validation results.

## Answer Validation

When you provide `expected_answer` in your questions, the explorer validates the agent's answer against yours using an LLM comparison. This appears in the CLI output as:

```
[1/12] ✓ Success (high confidence)
  Validation: full
```

**Match levels:**

| Match | Meaning |
|-------|---------|
| `full` | Agent's answer conveys the same information as expected (even if worded differently) |
| `partial` | Answer contains some expected information but is missing parts |
| `none` | Answer is different or contradictory |
| `gap_identified` | Agent correctly determined the data doesn't exist in this dataset |

Validation results flow into the final `exploration-results.md` — full matches produce [High] confidence patterns, partial matches produce [Medium], and failures are documented in the Known Limitations section.

## CLI Options

| Option | Description | Default |
|--------|-------------|---------|
| `--mcp-url <url>` | Agent Context server URL (required) | — |
| `--questions <path>` | Path to questions JSON file (required) | — |
| `--sanity-token <token>` | Sanity API read token for authentication | — |
| `--anthropic-api-key <key>` | Anthropic API key (or set `ANTHROPIC_API_KEY` env var) | — |
| `--model <model>` | Claude model for exploration | `claude-sonnet-4-20250514` |
| `--output <dir>` | Output directory | `./explorer-output-{timestamp}` |
| `--concurrency <n>` | Number of questions to explore in parallel | `3` |
| `--help` | Show help message | — |

## How It Works

1. **Connect** — Establishes connection to the Agent Context server and discovers available tools
2. **Explore** — For each question, an AI agent attempts to answer it using the available tools
3. **Learn** — The agent documents what worked, what failed, and what surprised it
4. **Synthesize** — All exploration logs are synthesized into a unified knowledge document

The key insight: **failures are more valuable than successes**. A naive agent can figure out what works through trial-and-error. What they can't discover is why something that looks right doesn't work, or why data that should exist is missing.

## Example Output

Here's a snippet from a generated `exploration-results.md`:

```markdown
## Schema Reference

| Document Type | Use For | Key Fields |
|---------------|---------|------------|
| product | Product info, specs | name, description, specs, variants |
| category | Product categorization | title, slug, products[] |
| support-article | Help content | title, body, relatedProducts[] |

## Query Patterns

### Product Details
**When to use:** User asks about a specific product's features or specifications

*[_type == "product" && name match $productName][0]{
  name, description, specs, variants
}

**Important:** Always use `name` not `title` — the `title` field is null on products.

### Product Comparison
**When to use:** User wants to compare two or more products

*[_type == "product" && name in $productNames]{
  name, specs, variants
}

**Important:** Filter results by locale if your dataset has multiple language variants.

## Critical Rules

- Always use `name` for product lookups — `title` is null on all product documents
- Always filter by locale when querying products to avoid duplicate results
- Never query `inventory` or `stockLevel` — these fields are always null (managed in external system)

## Known Limitations

- Inventory and stock data lives in Shopify, not this dataset [High confidence]
- Pricing data lives in Commerce API [High confidence]
- The `title` field on products is always null — use `name` instead [High confidence]

## Exploration Coverage

**Validated areas:** product specs, product comparison, category browsing, support content

**Confidence:** High — 12 questions explored with 92% success rate

**Not explored:** user reviews, order history, real-time inventory
```

## Using the Output

After running the explorer:

1. Open `exploration-results.md` in your output directory
2. Review the generated instructions — adjust if needed for your specific use case
3. Copy the entire contents into your **Agent Context Document's `instructions` field**

This gives your agent dataset-specific knowledge from day one.

## Requirements

- Node.js 20+
- An Anthropic API key (pass via `--anthropic-api-key` or set `ANTHROPIC_API_KEY` env var)
- A Sanity Agent Context server URL (see [Agent Context setup](https://github.com/sanity-io/agent-context))

## License

MIT