npm package: @sanity/ailf-studio
Description: AI Literacy Framework — Sanity Studio dashboard plugin
Latest version: 1.1.1

# @sanity/ailf-studio

> ⚠️ **Internal package.** This package is published publicly for convenience
> but is intended for internal Sanity use only. APIs and schemas may change
> without notice. No support is provided for external consumers.

Sanity Studio dashboard plugin for the **AI Literacy Framework**. Visualizes
evaluation reports, score trends, comparisons, and content impact — directly
inside Sanity Studio with no external backend.

All data is read from the Sanity Content Lake via GROQ.

## Installation

Install the plugin into any Sanity Studio that has access to the dataset where
AILF reports are stored.

### 1. Add the dependency

```bash
pnpm add @sanity/ailf-studio
```

#### Within the monorepo

```bash
pnpm add @sanity/ailf-studio@workspace:*
```

### 2. Register the plugin

The recommended approach registers both the document schemas and the dashboard
tool in one call:

```ts
// sanity.config.ts
import { defineConfig } from "sanity"
import { ailfPlugin } from "@sanity/ailf-studio"

export default defineConfig({
  // ... your existing config
  plugins: [
    ailfPlugin(),
    // ... other plugins
  ],
})
```

This registers:

- The `ailf.report` document type (read-only evaluation reports)
- The `ailf.webhookConfig` document type (webhook-triggered evaluation settings)
- The `ailf.task` document type (evaluation task definitions)
- The `ailf.featureArea` document type (feature area groupings)
- The `ailf.referenceSolution` document type (gold-standard reference
  implementations)
- The `ailf.evalRequest` document type (evaluation request triggers)
- The **AI Literacy Framework** dashboard tool in the Studio sidebar

#### Document Actions

The plugin registers two document actions for triggering evaluations directly
from Studio:

- **Run Task Eval** (on `ailf.task` documents) — evaluates a single task. Click
  ▶ in the document actions menu to run all test cases for the task against the
  current documentation. The button shows the score when complete (~10–15 min).
  No secrets needed — it creates an `ailf.evalRequest` document that a
  server-side webhook dispatches to the pipeline.

- **Run AI Eval** (on content releases) — evaluates all tasks affected by a
  content release. Appears in the release detail page's action bar. Answers "did
  my doc changes help or hurt AI agent performance?" Shows score and delta vs
  baseline when complete.

Both actions use the same mechanism: they create an `ailf.evalRequest` document
in the Content Lake with `status: "pending"`. A server-side Sanity webhook picks
up the document and dispatches the pipeline via GitHub Actions. The Studio
component polls for the resulting report and updates the button label with the
score.

### 3. Alternative: tool-only installation

If you only want the dashboard tool without the document schemas (e.g., the
schemas are already registered elsewhere):

```ts
// sanity.config.ts
import { defineConfig } from "sanity"
import { ailfTool } from "@sanity/ailf-studio"

export default defineConfig({
  // ... your existing config
  tools: [ailfTool()],
})
```

### 4. Alternative: schema-only installation

If you want the document schemas without the dashboard (e.g., to query reports
programmatically):

```ts
// sanity.config.ts
import { defineConfig } from "sanity"
import {
  reportSchema,
  webhookConfigSchema,
  taskSchema,
  featureAreaSchema,
  referenceSolutionSchema,
} from "@sanity/ailf-studio"

export default defineConfig({
  // ... your existing config
  schema: {
    types: [
      reportSchema,
      webhookConfigSchema,
      taskSchema,
      featureAreaSchema,
      referenceSolutionSchema,
    ],
  },
})
```

## Task Execution Workflows

Tasks created in Studio are automatically included in every pipeline run — no
registration step needed. There are four ways to execute tasks:

| Method                         | Trigger                                                           | Scope                         |
| ------------------------------ | ----------------------------------------------------------------- | ----------------------------- |
| **Run Task Eval** action       | Click ▶ on any `ailf.task` document                               | Single task                   |
| **Run AI Eval** release action | Click button on a content release page                            | Tasks affected by the release |
| **CLI pipeline**               | `ailf pipeline` (with optional `--area`/`--task`/`--tag` filters) | All enabled tasks             |
| **Scheduled pipeline**         | GitHub Actions cron (daily + weekly)                              | All enabled tasks             |

See the
[CONTRIBUTING_TASKS](https://github.com/sanity-labs/ai-literacy-framework/blob/main/docs/CONTRIBUTING_TASKS.md#running-your-task)
guide for the full execution flow and details on each method.

## Dashboard Views

The plugin provides three tab views plus a detail drill-down, accessible from
the **AI Literacy Framework** tool in the Studio sidebar.

### Latest Reports

A card list of the most recent evaluation reports. Each card shows:

- Overall score, doc lift, and lowest-scoring area
- Evaluation mode, source, and trigger type
- Git metadata (branch, PR number, origin repo) when available
- Auto-comparison delta against the previous run

Click any card to navigate to the Report Detail view.

The view includes a **search bar** for filtering reports by document slug, area,
or content release perspective.

### Score Timeline

A line chart of overall and per-area scores over time. Filterable by:

- **Source** — which documentation source was evaluated (e.g., production,
  branch deploy)
- **Mode** — evaluation mode (baseline, observed, agentic)

Data points are interactive — click to jump to the full report.

### Compare

Side-by-side comparison of any two reports. Select a baseline and experiment
report from dropdowns, then view:

- Overall score and doc-lift deltas
- Per-area deltas (improved / regressed / unchanged)
- Per-model deltas (when both reports include per-model breakdowns)
- Noise threshold classification

### Report Detail

Full drill-down into a single report (navigated from Latest Reports or a direct
URL):

- **Overview stats** — composite score, doc lift, cost, duration
- **Per-area score table** with all dimensions (task completion, code
  correctness, doc coverage, lift from docs)
- **Three-layer table** — floor / ceiling / actual decomposition (when
  available)
- **Per-model breakdowns** with cost-per-quality-point
- **Judgment list** — individual grader verdicts with reasoning
- **Recommendations** — gap analysis remediation suggestions (when available)
- **Provenance card** — trigger, git info (branch, PR, origin repo), grader
  model, context hash, eval fingerprint
- **Auto-comparison summary** against the previous comparable run

## Filtering

The Dashboard and Score Timeline views share global filters:

- **Source filter** — values are auto-populated from distinct
  `provenance.source.name` values across all reports
- **Mode filter** — values are auto-populated from distinct `provenance.mode`
  values

Filters are applied via GROQ query parameters, so only matching reports are
fetched.

## Dataset Configuration

The plugin reads reports from whatever dataset the Studio is configured to use.
To point it at a dedicated report dataset, configure the Studio's dataset:

```ts
export default defineConfig({
  projectId: "<your-project-id>",
  dataset: "my-report-dataset",
  plugins: [ailfPlugin()],
})
```

Reports are written by the evaluation pipeline (`ailf pipeline --publish`).

## Exported API

The plugin exports building blocks for custom views or extensions.

### Plugin & Tool

| Export       | Description                  |
| ------------ | ---------------------------- |
| `ailfPlugin` | Full plugin (schemas + tool) |
| `ailfTool`   | Dashboard tool only          |

### Schemas

| Export                    | Description                                       |
| ------------------------- | ------------------------------------------------- |
| `reportSchema`            | `ailf.report` document type definition            |
| `webhookConfigSchema`     | `ailf.webhookConfig` document type                |
| `taskSchema`              | `ailf.task` document type definition              |
| `featureAreaSchema`       | `ailf.featureArea` document type definition       |
| `referenceSolutionSchema` | `ailf.referenceSolution` document type definition |
| `evalRequestSchema`       | `ailf.evalRequest` document type definition       |

### Components

| Export              | Description                                                                                   |
| ------------------- | --------------------------------------------------------------------------------------------- |
| `AssertionInput`    | Custom input for task assertions with contextual type descriptions and monospace code styling |
| `CanonicalDocInput` | Custom input for canonical doc references with polymorphic resolution type help               |
| `ReleasePicker`     | Content release perspective picker for evaluation scoping                                     |
| `MirrorBanner`      | Banner showing repo source, sync status, and provenance for mirrored tasks                    |
| `SyncStatusBadge`   | Colored badge (green/yellow/red) showing sync freshness of mirrored tasks                     |

### Document Actions

| Export                      | Description                                                                                  |
| --------------------------- | -------------------------------------------------------------------------------------------- |
| `GraduateToNativeAction`    | Converts a mirrored (read-only) task to a native (editable) task by removing origin          |
| `RunTaskEvaluationAction`   | Triggers a pipeline evaluation scoped to a single task (registered on `ailf.task` documents) |
| `createRunEvaluationAction` | Factory for creating a Studio action that triggers release-scoped evaluations                |

### Glossary

| Export     | Description                                                              |
| ---------- | ------------------------------------------------------------------------ |
| `GLOSSARY` | Centralized tooltip descriptions for all evaluation metrics and concepts |

### GROQ Queries

| Export                         | Description                                |
| ------------------------------ | ------------------------------------------ |
| `latestReportsQuery`           | N most recent reports (filterable)         |
| `scoreTimelineQuery`           | Score data points over time                |
| `reportDetailQuery`            | Full report with all fields                |
| `comparisonPairQuery`          | Two reports for side-by-side comparison    |
| `contentImpactQuery`           | Reports related to a document ID           |
| `recentDocumentEvalsQuery`     | Recent evaluations for a specific document |
| `articleSearchQuery`           | Full-text search across article documents  |
| `distinctSourcesQuery`         | All unique source names                    |
| `distinctModesQuery`           | All unique evaluation modes                |
| `distinctAreasQuery`           | All unique feature areas                   |
| `distinctModelsQuery`          | All unique model identifiers               |
| `distinctPerspectivesQuery`    | All unique content release perspectives    |
| `distinctTargetDocumentsQuery` | All unique target document slugs           |

### Types

| Export                       | Description                                                           |
| ---------------------------- | --------------------------------------------------------------------- |
| `ReportListItem`             | Shape returned by `latestReportsQuery`                                |
| `ReportDetail`               | Shape returned by `reportDetailQuery`                                 |
| `TimelineDataPoint`          | Shape returned by `scoreTimelineQuery`                                |
| `ComparisonData`             | Auto-comparison data embedded in reports                              |
| `ContentImpactItem`          | Shape returned by `contentImpactQuery`                                |
| `ProvenanceData`             | Report provenance metadata                                            |
| `SummaryData`                | Score summary (overall + per-area + per-model)                        |
| `ScoreItem`                  | Individual area score entry                                           |
| `RecommendationGap`          | Single gap analysis recommendation                                    |
| `RecommendationsData`        | Full recommendations payload                                          |
| `JudgmentData`               | Individual grader judgment with reasoning                             |
| `DocumentRef`                | Canonical document reference (re-exported from `@sanity/ailf-shared`) |
| `ScoreGrade`                 | Letter grade type (re-exported from `@sanity/ailf-shared`)            |
| `scoreGrade`                 | Function to compute letter grade from numeric score                   |
| `RunEvaluationActionOptions` | Options for `createRunEvaluationAction` factory                       |

### Utility Functions

| Export               | Description                                               |
| -------------------- | --------------------------------------------------------- |
| `formatPercent`      | Format a number as a percentage string                    |
| `formatRelativeTime` | Format an ISO timestamp as relative time (e.g., "2h ago") |
| `formatDelta`        | Format a score delta with +/− sign                        |
| `formatDuration`     | Format milliseconds as human-readable duration            |

## Development

```bash
# Build the plugin
pnpm --filter @sanity/ailf-studio build

# Watch mode (rebuilds on file changes)
pnpm --filter @sanity/ailf-studio dev

# Build everything (from repo root)
turbo build
```

The plugin uses [tsup](https://github.com/egoist/tsup) for bundling. The
consuming Studio's bundler (Vite) handles the final bundle.

## Related Documentation

- [Report Store Design](https://github.com/sanity-labs/ai-literacy-framework/blob/main/docs/design-docs/report-store/index.md)
  — full architecture and implementation plan
- [Visibility & Workflows](https://github.com/sanity-labs/ai-literacy-framework/blob/main/docs/design-docs/report-store/visibility-workflows.md)
  — design rationale for the dashboard views
- [Report Store Architecture](https://github.com/sanity-labs/ai-literacy-framework/blob/main/docs/design-docs/report-store/architecture.md)
  — Sanity Content Lake as the system of record