Skill: geminimir/stripemeter (version 0.1.0)

--- name: stripemeter description: Integrate Stripe usage-based billing with idempotent event ingestion, late-event handling, and pre-invoice reconciliation. Use when implementing usage metering, tracking API calls or seats, pushing usage to Stripe, handling billing drift, or building usage-based pricing. --- # StripeMeter StripeMeter is a Stripe-native usage metering system that ensures correct usage totals for usage-based billing. It dedupes retries, handles late events with watermarks, keeps running counters, and pushes only deltas to Stripe. ## Quick Start ```bash git clone https://github.com/geminimir/stripemeter && cd stripemeter cp .env.example .env && docker compose up -d && pnpm -r build pnpm db:migrate && pnpm dev ``` ## Core Concepts ### Events (Immutable Ledger) Every usage event stored with deterministic idempotency key. Events are never deleted or modified. ### Counters (Materialized Aggregations) Pre-computed aggregations (sum/max/last) by tenant, metric, customer, and period. Updated in near-real-time. ### Watermarks (Late Event Handling) Each counter maintains a watermark timestamp. Events within lateness window (default 48h) trigger re-aggregation. ### Delta Push (Stripe Synchronization) Tracks `pushed_total` per subscription item and only sends delta to Stripe. ## API Endpoints ### Ingest Events ```bash curl -X POST http://localhost:3000/v1/events/ingest \ -H "Content-Type: application/json" \ -d '{ "events": [{ "tenantId": "your-tenant-id", "metric": "api_calls", "customerRef": "cus_ABC123", "quantity": 100, "ts": "2025-01-16T14:30:00Z" }] }' ``` ### Get Cost Projection ```bash curl -X POST http://localhost:3000/v1/usage/projection \ -H "Content-Type: application/json" \ -d '{"tenantId": "your-tenant-id", "customerRef": "cus_ABC123"}' ``` ### Health & Metrics - Readiness: `GET /health/ready` - Metrics: `GET /metrics` - Events: `GET /v1/events?tenantId=X&limit=10` ## Node.js SDK ```typescript import { createClient } from '@stripemeter/sdk-node'; const client = createClient({ apiUrl: 'http://localhost:3000', tenantId: 'your-tenant-id', customerId: 'cus_ABC123' }); // Track usage await client.track({ metric: 'api_calls', customerRef: 'cus_ABC123', quantity: 100, meta: { endpoint: '/v1/search' } }); // Get live usage const usage = await client.getUsage('cus_ABC123'); const projection = await client.getProjection('cus_ABC123'); ``` ## Python SDK ```python from stripemeter import StripeMeterClient client = StripeMeterClient( api_url="http://localhost:3000", tenant_id="your-tenant-id", customer_id="cus_ABC123" ) client.track( metric="api_calls", customer_ref="cus_ABC123", quantity=100 ) ``` ## Stripe Billing Driver For direct Stripe integration, use the stripe-driver package: ```typescript import { StripeBillingDriverImpl } from '@stripemeter/stripe-driver'; const driver = new StripeBillingDriverImpl({ liveKey: process.env.STRIPE_SECRET_KEY, testKey: process.env.STRIPE_TEST_SECRET_KEY }); // Record usage to Stripe await driver.recordUsage({ mode: 'live', stripeAccount: 'default', subscriptionItemId: 'si_xxx', quantity: 100, periodStart: '2025-01-01', idempotencyKey: 'unique-key' }); // Get usage summary const summary = await driver.getUsageSummary( 'si_xxx', '2025-01-01', 'default' ); ``` ## Shadow Mode Test Stripe usage posting without affecting live invoices: 1. Set `STRIPE_TEST_SECRET_KEY` in environment 2. Mark price mapping with `shadow=true` 3. Provide `shadowStripeAccount`, `shadowPriceId` 4. Live invoices remain unaffected ## Pricing Simulator ```typescript import { InvoiceSimulator } from '@stripemeter/pricing-lib'; const simulator = new InvoiceSimulator(); const result = simulator.simulate({ customerId: 'test', periodStart: '2024-01-01', periodEnd: '2024-02-01', usageItems: [{ metric: 'api_calls', quantity: 25000, priceConfig: { model: 'tiered', currency: 'USD', tiers: [ { upTo: 10000, unitPrice: 0.01 }, { upTo: 50000, unitPrice: 0.008 }, { upTo: null, unitPrice: 0.005 } ] } }] }); ``` ## Project Structure ``` stripemeter/ ├── packages/ │ ├── core/ # Shared types, schemas │ ├── database/ # Drizzle ORM + Redis │ ├── pricing-lib/ # Pricing calculator │ ├── stripe-driver/ # Direct Stripe API driver │ ├── sdk-node/ # Node.js SDK │ └── sdk-python/ # Python SDK ├── apps/ │ ├── api/ # REST API (Fastify) │ ├── workers/ # Background workers (BullMQ) │ ├── admin-ui/ # Admin dashboard │ └── customer-widget/# Embeddable widget ``` ## Environment Variables ```bash STRIPE_SECRET_KEY=sk_live_xxx # Live Stripe key STRIPE_TEST_SECRET_KEY=sk_test_xxx # Test Stripe key (for shadow mode) DATABASE_URL=postgres://... # PostgreSQL connection REDIS_URL=redis://... # Redis connection ``` ## Common Tasks ### Verify Idempotency ```bash # Send same event twice - counts once TENANT_ID=$(uuidgen) bash examples/api-calls/send.sh curl http://localhost:3000/metrics | grep ingest ``` ### Run Reconciliation Check drift between local totals and Stripe: - Differences beyond 0.5% epsilon trigger investigation - See [RECONCILIATION.md](RECONCILIATION.md) for runbook ### Replay Late Events ```bash curl -X POST http://localhost:3000/v1/replay \ -H "Content-Type: application/json" \ -d '{"tenantId": "X", "dryRun": true}' ``` ## Additional Resources - [API Documentation](docs/welcome.md) - [Simulator Guide](docs/simulator-getting-started.md) - [Reconciliation Runbook](RECONCILIATION.md) - [Alert Configuration](ops/ALERTS.md)

path: RECONCILIATION.md
content: # Reconciliation Runbook (Operator Playbook)

Concise, copy-pasteable steps to investigate drift and repair discrepancies between StripeMeter and Stripe. Runnable against local docker-compose.

## Policy

- Drift epsilon: 0.5% relative difference per subscription item per period
- Within epsilon (≤ 0.5%): record a small adjustment to align and proceed
- Beyond epsilon (> 0.5%): backfill late/missing events, then re-run aggregation

## Core mechanics (how the system behaves)

- Watermarks: each counter tracks a last-processed timestamp; late events within the window trigger re-aggregation
- Delta push: writer sends only the delta from `pushed_total` per Stripe item (idempotent)
- Reconciliation loop: periodically compares local totals vs Stripe reported usage and flags items beyond epsilon

---

## Triage Runbook

### 0) Prerequisites (local)

```bash
# Bring up infra
docker compose -f docker-compose.yml up -d

# Start app stack (in another terminal)
pnpm -r build && pnpm dev
```

Optional monitoring stack:

```bash
docker compose -f docker-compose.prod.yml --profile monitoring up -d
# Prometheus: http://localhost:${PROMETHEUS_PORT:-9090}
# Grafana:    http://localhost:${GRAFANA_PORT:-3001}  (admin/admin by default)
```

### 1) Verify service health and scrape status

```bash
# API health
curl -fsS http://localhost:3000/health/ready | jq .

# API metrics are exposed for Prometheus
curl -fsS http://localhost:3000/metrics | head -n 20

# If monitoring profile is enabled, check Prometheus targets
open "http://localhost:9090/targets" || echo "Open http://localhost:9090/targets"
```

Expected: `/health/ready` returns `healthy` or `degraded`; `/metrics` includes `http_requests_total` and `http_request_duration_seconds_bucket`.

### 2) Inspect drift metrics and logs

Prometheus queries (paste into Prometheus/Grafana):

```
# Percentage drift by item and period
max_over_time(reconciliation_diff_pct[15m])

# Absolute drift by item and period
max_over_time(reconciliation_diff_abs[15m])

# Reconciliation run cadence and latency
increase(recon_runs_total[1h])
histogram_quantile(0.95, sum(rate(recon_duration_seconds_bucket[5m])) by (le))

# Ingest p95 latency for /v1/events/ingest
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{route="/v1/events/ingest"}[5m])))

# HTTP 5xx rate by route
sum by (route) (rate(http_requests_total{status_code=~"5.."}[5m]))
```

Logs (newest last):

```bash
docker logs -n 200 -f stripemeter-api | sed -n '1,120p'
docker logs -n 200 -f stripemeter-workers | sed -n '1,120p'
```

### 3) Identify the affected item

Gather: `tenantId`, `customerRef`, `metric`, and billing period.

```bash
# Quick period summary (demo endpoint)
curl -s "http://localhost:3000/v1/reconciliation/summary?tenantId=demo&metric=requests" | jq .
```

### 4) Re-aggregate a window (safe retry/backoff)

Use the Replay API to recompute counters over a time window. Always run a dry-run first.

```bash
# Dry-run last 24h for one metric
curl -s -X POST http://localhost:3000/v1/replay \
  -H 'Content-Type: application/json' \
  -d '{
    "tenantId": "demo",
    "metrics": ["requests"],
    "since": "-PT24H",
    "until": "now",
    "mode": "dry-run"
  }' | jq .

# Apply if the dry-run looks correct (idempotent)
curl -s -X POST http://localhost:3000/v1/replay \
  -H 'Content-Type: application/json' \
  -d '{
    "tenantId": "demo",
    "metrics": ["requests"],
    "since": "-PT24H",
    "until": "now",
    "mode": "apply"
  }' | jq .
```

Guidance:
- Safe to retry with exponential backoff (e.g., 5s, 15s, 30s) if queues are busy; the writer is delta/idempotent
- Expected time: re-aggregation of ~10k late events ≤ 2 s on a laptop

### 5) Targeted replays

Narrow the blast radius by period or customer where needed.

```bash
# Replay a specific period
curl -s -X POST http://localhost:3000/v1/replay \
  -H 'Content-Type: application/json' \
  -d '{
    "tenantId": "demo",
    "metrics": ["requests"],
    "since": "2025-01-01T00:00:00Z",
    "until": "2025-02-01T00:00:00Z",
    "mode": "dry-run"
  }' | jq .
```

### 6) Confirm resolution

```bash
# Reconciliation summary should show drift back within epsilon
curl -s "http://localhost:3000/v1/reconciliation/summary?tenantId=demo&metric=requests" | jq .

# Optionally trigger a reconciliation cycle
curl -s -X POST http://localhost:3000/v1/reconciliation/run -H 'Content-Type: application/json' \
  -d '{"tenantId":"demo"}' | jq .
```

If still beyond epsilon: inspect raw events and watermarks, then repeat step 4 with a wider window.

---

## Common pitfalls

- Tenant or metric mismatch (e.g., wrong `tenantId` or unconfigured metric mapping)
- Timezone boundaries causing the wrong billing period window
- Events arriving after the lateness window → require adjustments instead of automatic re-aggregation
- Missing Stripe secrets or permissions block writer parity
- Placeholder drift gauges (`reconciliation_diff_*`) not yet wired in your environment

## Examples

- Late event (< 48h): falls within lateness window, re-aggregate; if residual diff ≤ 0.5%, adjust.
- Duplicate event: idempotency key should dedupe; if not, create a negative adjustment.

## References & dashboards

- Health and metrics: `GET /health/ready`, `GET /metrics`
- Replay API: `POST /v1/replay` (dry-run → apply)
- Reconciliation: `GET /v1/reconciliation/summary`, `POST /v1/reconciliation/run`
- Alerts and “what to do next”: see `ops/ALERTS.md`


path: docs/api/ingest.md
content: # API — Ingest (v0.3.0)

[← Back to Welcome](../welcome.md)

POST /v1/events/ingest

Headers
- Content-Type: application/json
- Idempotency-Key (string, optional): Custom idempotency key for the entire batch
  - Precedence: `event.idempotencyKey` in the body takes priority over the header. If the body event lacks `idempotencyKey`, the header value is applied. If neither are provided, the server deterministically generates one.

Request Body
```json
{
  "events": [
    {
      "tenantId": "demo",
      "metric": "requests",
      "customerRef": "cus_123",
      "resourceId": "res_1",
      "quantity": 100,
      "ts": "2025-01-16T14:30:00Z",
      "source": "api",
      "meta": {
        "endpoint": "/v1/search",
        "region": "us-east-1"
      }
    }
  ]
}
```

Response (per-event results)
```json
{
  "accepted": 1,
  "duplicates": 0,
  "requestId": "req-123",
  "results": [
    {
      "idempotencyKey": "evt_abc123",
      "status": "accepted"
    }
  ],
  "errors": []
}
```

Examples
```bash
# Send a single event with idempotency
curl -X POST http://localhost:3000/v1/events/ingest \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: my-batch-123" \
  -d '{
    "events": [{
      "tenantId": "demo",
      "metric": "requests",
      "customerRef": "cus_123",
      "quantity": 100,
      "ts": "2025-01-16T14:30:00Z"
    }]
  }'

# Send the same event again (should be deduplicated)
curl -X POST http://localhost:3000/v1/events/ingest \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: my-batch-123" \
  -d '{
    "events": [{
      "tenantId": "demo",
      "metric": "requests",
      "customerRef": "cus_123",
      "quantity": 100,
      "ts": "2025-01-16T14:30:00Z"
    }]
  }'
```

Notes
- Events with the same idempotency key are counted exactly once
- Precedence: body `idempotencyKey` > `Idempotency-Key` header > server-generated key
- If no `Idempotency-Key` header or body key is provided, one is generated from the event fields
- The `ts` field should be in ISO 8601 format
- `meta` can contain any additional JSON data for the event
- Batch size limit: 1000 events per request


path: docs/api/reconciliation-summary.md
content: # API — Reconciliation Summary (v0.3.0)

[← Back to Welcome](../welcome.md)

GET /v1/reconciliation/summary

Query Parameters
- `tenantId` (string, required): Tenant ID to get reconciliation summary for
- `metric` (string, required): Metric name to check reconciliation for
- `customerRef` (string, optional): Filter by specific customer
- `period` (string, optional): Time period (default: current billing period)

Response
```json
{
  "tenantId": "demo",
  "metric": "requests",
  "period": {
    "start": "2025-01-01T00:00:00Z",
    "end": "2025-02-01T00:00:00Z"
  },
  "reconciliation": {
    "localTotal": 15500,
    "stripeTotal": 15450,
    "drift": 50,
    "driftPercentage": 0.32,
    "status": "drift_detected",
    "lastChecked": "2025-01-16T14:30:00Z"
  },
  "counters": {
    "beforeReplay": {
      "total": 15450,
      "byCustomer": {
        "cus_123": 8200,
        "cus_456": 7250
      }
    },
    "afterReplay": {
      "total": 15500,
      "byCustomer": {
        "cus_123": 8250,
        "cus_456": 7250
      }
    }
  },
  "events": {
    "totalProcessed": 15500,
    "lateEvents": 12,
    "duplicatesFiltered": 8,
    "lastEventTime": "2025-01-16T14:25:00Z"
  }
}
```

Status Values
- `parity`: Local and Stripe totals match (drift ≤ 0.5%)
- `drift_detected`: Drift detected but within tolerance
- `drift_critical`: Drift exceeds critical threshold (> 5%)
- `reconciliation_needed`: Manual intervention required

Examples
```bash
# Get reconciliation summary for requests metric
curl -s "http://localhost:3000/v1/reconciliation/summary?tenantId=demo&metric=requests" | jq

# Get summary for specific customer
curl -s "http://localhost:3000/v1/reconciliation/summary?tenantId=demo&metric=requests&customerRef=cus_123" | jq

# Get summary for specific period
curl -s "http://localhost:3000/v1/reconciliation/summary?tenantId=demo&metric=requests&period=2025-01" | jq
```

Notes
- Drift absolute is `|local - stripe|`; drift percentage is `drift_abs / stripe` with 1.0 when `stripe == 0 && drift_abs > 0`.
- CSV export: add `format=csv` query param. A totals row is appended as `TOTAL`.
- All timestamps use UTC.


path: docs/api/usage-history.md
content: # API — Usage History (v0.3.0)

[← Back to Welcome](../welcome.md)

GET /v1/usage/history

Query Parameters
- `tenantId` (string, required)
- `metric` (string, required)
- `customerRef` (string, optional)
- `since` (ISO 8601 or relative, optional)
- `until` (ISO 8601 or "now", optional)
- `interval` (string, optional; one of minute|hour|day; default day)

Response
```json
{
  "tenantId": "demo",
  "metric": "requests",
  "timeRange": {
    "since": "2025-01-15T00:00:00Z",
    "until": "2025-01-16T00:00:00Z",
    "interval": "hour"
  },
  "series": [
    { "ts": "2025-01-15T00:00:00Z", "value": 120 },
    { "ts": "2025-01-15T01:00:00Z", "value": 98 }
  ],
  "totals": {
    "value": 5230
  }
}
```

Examples
```bash
# Hourly time-series plus totals
curl -s "http://localhost:3000/v1/usage/history?tenantId=demo&metric=requests&since=-P1D&interval=hour" | jq
```

Notes
- Use this alongside Replay to visualize before/after counters
- `interval` controls aggregation bucket size


path: docs/simulator-getting-started.md
content: # StripeMeter Pricing Simulator - Getting Started

The StripeMeter Pricing Simulator is a powerful tool that helps you test different pricing strategies, validate billing logic, and optimize your revenue models before deploying to production. This guide will help you understand how to use the simulator effectively.

## What is the Pricing Simulator?

The simulator allows you to:
- **Test pricing models** (tiered, volume, graduated) with realistic usage data
- **Compare revenue outcomes** between different pricing strategies
- **Validate billing accuracy** before customers see their bills
- **Simulate complex scenarios** including credits, commitments, and adjustments
- **Optimize pricing** for different customer segments

## Quick Start

### 1. Basic Invoice Simulation

```typescript
import { InvoiceSimulator } from '@stripemeter/pricing-lib';

const simulator = new InvoiceSimulator();

// Define your usage and pricing
const simulationInput = {
  customerId: 'cust_123',
  periodStart: '2024-01-01',
  periodEnd: '2024-02-01',
  usageItems: [
    {
      metric: 'api_calls',
      quantity: 15000,
      priceConfig: {
        model: 'tiered',
        currency: 'USD',
        tiers: [
          { upTo: 1000, unitPrice: 0.10 },    // $0.10 for first 1,000
          { upTo: 10000, unitPrice: 0.08 },   // $0.08 for next 9,000
          { upTo: null, unitPrice: 0.05 }     // $0.05 for everything above
        ]
      }
    }
  ]
};

// Run the simulation
const invoice = simulator.simulate(simulationInput);
console.log(`Total bill: $${invoice.total}`);
```

### 2. Compare Pricing Models

```typescript
const usage = 15000; // API calls

const pricingModels = {
  tiered: {
    model: 'tiered',
    currency: 'USD',
    tiers: [
      { upTo: 1000, unitPrice: 0.10 },
      { upTo: 10000, unitPrice: 0.08 },
      { upTo: null, unitPrice: 0.05 }
    ]
  },
  volume: {
    model: 'volume',
    currency: 'USD',
    tiers: [
      { upTo: 1000, unitPrice: 0.10 },
      { upTo: 10000, unitPrice: 0.08 },
      { upTo: null, unitPrice: 0.05 }
    ]
  },
  graduated: {
    model: 'graduated',
    currency: 'USD',
    tiers: [
      { upTo: 1000, flatPrice: 50, unitPrice: 0.05 },
      { upTo: 10000, flatPrice: 100, unitPrice: 0.03 },
      { upTo: null, flatPrice: 200, unitPrice: 0.01 }
    ]
  }
};

// Compare results
Object.entries(pricingModels).forEach(([modelName, config]) => {
  const result = simulator.simulate({
    customerId: 'cust_123',
    periodStart: '2024-01-01',
    periodEnd: '2024-02-01',
    usageItems: [{ metric: 'api_calls', quantity: usage, priceConfig: config }]
  });
  console.log(`${modelName}: $${result.total}`);
});
```

## Pricing Model Types

### Tiered Pricing
Each unit is priced according to the tier it falls into.

```typescript
const tieredConfig = {
  model: 'tiered',
  currency: 'USD',
  tiers: [
    { upTo: 1000, unitPrice: 0.10 },    // First 1,000 units at $0.10
    { upTo: 10000, unitPrice: 0.08 },   // Next 9,000 units at $0.08
    { upTo: null, unitPrice: 0.05 }     // All remaining at $0.05
  ]
};

// 15,000 units = (1,000 × $0.10) + (9,000 × $0.08) + (5,000 × $0.05) = $1,070
```

### Volume Pricing
All units are priced at the rate of the tier they reach.

```typescript
const volumeConfig = {
  model: 'volume',
  currency: 'USD',
  tiers: [
    { upTo: 1000, unitPrice: 0.10 },
    { upTo: 10000, unitPrice: 0.08 },
    { upTo: null, unitPrice: 0.05 }
  ]
};

// 15,000 units = 15,000 × $0.05 = $750 (lowest applicable rate)
```

### Graduated Pricing
Each tier has a flat fee plus a per-unit rate.

```typescript
const graduatedConfig = {
  model: 'graduated',
  currency: 'USD',
  tiers: [
    { upTo: 1000, flatPrice: 50, unitPrice: 0.05 },
    { upTo: 10000, flatPrice: 100, unitPrice: 0.03 },
    { upTo: null, flatPrice: 200, unitPrice: 0.01 }
  ]
};

// 15,000 units = $200 + (15,000 × $0.01) = $350
```

## Advanced Features

### Credits and Commitments

```typescript
const simulationWithCredits = {
  customerId: 'cust_123',
  periodStart: '2024-01-01',
  periodEnd: '2024-02-01',
  usageItems: [
    {
      metric: 'api_calls',
      quantity: 15000,
      priceConfig: tieredConfig
    }
  ],
  credits: [
    { amount: 100, reason: 'Welcome credit' },
    { amount: 50, reason: 'Referral bonus', expiresAt: '2024-03-01' }
  ],
  commitments: [
    { amount: 500, startDate: '2024-01-01', endDate: '2024-12-31', applied: 0 }
  ],
  taxRate: 8.5 // 8.5% tax
};

const invoice = simulator.simulate(simulationWithCredits);
console.log(`Subtotal: $${invoice.subtotal}`);
console.log(`Credits: -$${invoice.credits}`);
console.log(`Tax: $${invoice.tax}`);
console.log(`Total: $${invoice.total}`);
```

### Multi-Metric Billing

```typescript
const multiMetricSimulation = {
  customerId: 'enterprise_customer',
  periodStart: '2024-01-01',
  periodEnd: '2024-02-01',
  usageItems: [
    {
      metric: 'api_calls',
      quantity: 50000,
      priceConfig: {
        model: 'tiered',
        currency: 'USD',
        tiers: [
          { upTo: 10000, unitPrice: 0.01 },
          { upTo: 100000, unitPrice: 0.008 },
          { upTo: null, unitPrice: 0.005 }
        ]
      }
    },
    {
      metric: 'data_processed_gb',
      quantity: 500,
      priceConfig: {
        model: 'volume',
        currency: 'USD',
        tiers: [
          { upTo: 100, unitPrice: 0.50 },
          { upTo: 1000, unitPrice: 0.40 },
          { upTo: null, unitPrice: 0.30 }
        ]
      }
    },
    {
      metric: 'storage_gb_hours',
      quantity: 8760, // 1 GB for full month
      priceConfig: {
        model: 'flat',
        currency: 'USD',
        unitPrice: 0.02
      }
    }
  ]
};

const invoice = simulator.simulate(multiMetricSimulation);
console.log('Invoice breakdown:');
invoice.lineItems.forEach(item => {
  console.log(`${item.metric}: ${item.quantity} units × $${item.unitPrice} = $${item.subtotal}`);
});
console.log(`Total: $${invoice.total}`);
```

## Real-World Scenarios

See [Pricing Scenarios](./simulator-scenarios.md) for detailed examples including:
- **SaaS API Company** - tiered pricing for different customer segments
- **Data Processing Service** - volume-based pricing with commitments
- **Storage Provider** - graduated pricing with minimum charges
- **Enterprise Platform** - complex multi-metric billing

## Integration with StripeMeter

The simulator uses the same pricing calculations as the live StripeMeter system, ensuring:
- **Billing accuracy** - what you simulate is what customers pay
- **Parity testing** - validate against Stripe invoices
- **Confidence** - deploy pricing changes with certainty

### Testing Against Live Data

```typescript
// Use real usage data from your system
const realUsageData = await getCustomerUsage('cust_123', '2024-01-01', '2024-02-01');

const simulatedInvoice = simulator.simulate({
  customerId: 'cust_123',
  periodStart: '2024-01-01',
  periodEnd: '2024-02-01',
  usageItems: realUsageData.map(usage => ({
    metric: usage.metric,
    quantity: usage.total,
    priceConfig: getPricingConfig(usage.metric)
  }))
});

// Compare with actual Stripe invoice
const stripeInvoice = await stripe.invoices.retrieve('in_...');
const difference = Math.abs(simulatedInvoice.total - stripeInvoice.total);
console.log(`Difference: $${difference} (${(difference/stripeInvoice.total*100).toFixed(2)}%)`);
```

## Best Practices

### 1. Start Simple
- Begin with single-metric, single-tier pricing
- Add complexity gradually as you understand the patterns

### 2. Test Edge Cases
```typescript
const edgeCases = [
  { name: 'Zero usage', quantity: 0 },
  { name: 'Minimum tier', quantity: 1 },
  { name: 'Tier boundary', quantity: 1000 },
  { name: 'High usage', quantity: 1000000 }
];

edgeCases.forEach(testCase => {
  const result = simulator.simulate({
    customerId: testCase.name,
    periodStart: '2024-01-01',
    periodEnd: '2024-02-01',
    usageItems: [{ 
      metric: 'api_calls', 
      quantity: testCase.quantity, 
      priceConfig: tieredConfig 
    }]
  });
  console.log(`${testCase.name}: $${result.total}`);
});
```

### 3. Validate Revenue Impact
```typescript
// Test pricing changes against historical data
const historicalUsage = getHistoricalUsage('last_3_months');
const currentRevenue = calculateRevenue(historicalUsage, currentPricing);
const newRevenue = calculateRevenue(historicalUsage, proposedPricing);

console.log(`Revenue impact: ${((newRevenue - currentRevenue) / currentRevenue * 100).toFixed(1)}%`);
```

### 4. Customer Segment Analysis
```typescript
const segments = ['startup', 'growth', 'enterprise'];

segments.forEach(segment => {
  const segmentUsage = getSegmentUsage(segment);
  const avgBill = calculateAverageBill(segmentUsage, pricingConfig);
  console.log(`${segment}: Average bill $${avgBill}`);
});
```

## Next Steps

1. **Explore Scenarios** - Check out [detailed pricing scenarios](./simulator-scenarios.md)
2. **Set up CI Testing** - Automate pricing validation in your deployment pipeline
3. **Monitor in Production** - Use reconciliation reports to ensure accuracy
4. **Optimize Continuously** - Regular simulation helps optimize pricing strategy

## Need Help?

- 📖 [Full Documentation](../README.md)
- 🐛 [Report Issues](https://github.com/geminimir/stripemeter/issues)
- 💬 [Community Discussions](https://github.com/geminimir/stripemeter/discussions)
- 📧 [Support](mailto:[REDACTED_SECRET])

## Simulator CLI (sim) quickstart

Use the local CLI to validate scenarios, run them offline using the pricing engine, and report diffs against expected artifacts.

```bash
# Build workspace (first time or after changes)
pnpm build

# Validate all scenarios under examples/
pnpm run sim validate --dir examples

# Run scenarios and write results to results/
pnpm run sim run --dir examples --out results --seed 42

# Record the current results as expected artifacts next to scenarios
pnpm run sim run --dir examples --out results --record

# Diff actual results vs expected and fail on differences
pnpm run sim report --dir examples --results results --fail-on-diff
```

- Scenarios live in `scenarios/` or any folder you specify (see `examples/`).
- Scenario files use `.sim.json` extension and minimally require `metadata`, `inputs.usageItems`, and an `expected.total` to assert.
- You can set per-scenario tolerances via `tolerances.absolute` and `tolerances.relative` for numeric diffs.

Example scenario skeleton:

```json
{
  "metadata": { "name": "tiered-basic", "version": "1" },
  "inputs": {
    "customerId": "cust_1",
    "periodStart": "2024-01-01",
    "periodEnd": "2024-01-31",
    "usageItems": [
      {
        "metric": "requests",
        "quantity": 350,
        "priceConfig": {
          "model": "tiered",
          "currency": "USD",
          "tiers": [
            { "upTo": 100, "unitPrice": 0.1 },
            { "upTo": 500, "unitPrice": 0.08 },
            { "upTo": null, "unitPrice": 0.05 }
          ]
        }
      }
    ]
  },
  "expected": { "total": 28.0 },
  "tolerances": { "absolute": 0.001, "relative": 0.0005 }
}
```

CI runs validate → run → report for `examples/`. Place your own scenarios in `scenarios/` and mirror the same commands locally.


path: docs/simulator-scenarios.md
content: # StripeMeter Pricing Simulator - Real-World Scenarios

This guide provides realistic pricing scenarios that demonstrate how to use the StripeMeter simulator for different business models and use cases.

## Scenario 1: SaaS API Company - CloudAPI

**Business Model:** REST API service with tiered pricing for different customer segments.

### Customer Segments
- **Startup**: Free tier with overage charges
- **Growth**: Pro plan with included usage + overages  
- **Enterprise**: High-volume with committed spend

### Pricing Configuration

```typescript
import { InvoiceSimulator } from '@stripemeter/pricing-lib';

const simulator = new InvoiceSimulator();

// Startup tier - Free with overage
const startupPricing = {
  model: 'tiered',
  currency: 'USD',
  tiers: [
    { upTo: 1000, unitPrice: 0.00 },    // Free tier: 1,000 calls
    { upTo: 10000, unitPrice: 0.02 },   // Overage: $0.02 per call
    { upTo: null, unitPrice: 0.015 }    // High volume discount
  ]
};

// Growth tier - Pro plan
const growthPricing = {
  model: 'tiered', 
  currency: 'USD',
  tiers: [
    { upTo: 10000, unitPrice: 0.00 },   // Included in $29 plan
    { upTo: 100000, unitPrice: 0.01 },  // Overage: $0.01 per call
    { upTo: null, unitPrice: 0.008 }    // High volume discount
  ]
};

// Enterprise tier - Volume pricing with commitment
const enterprisePricing = {
  model: 'volume',
  currency: 'USD', 
  tiers: [
    { upTo: 100000, unitPrice: 0.008 },
    { upTo: 500000, unitPrice: 0.006 },
    { upTo: null, unitPrice: 0.004 }
  ]
};
```

### Usage Scenarios

```typescript
const scenarios = [
  {
    name: 'Startup - Light Usage',
    segment: 'startup',
    usage: 500,
    pricing: startupPricing,
    expected: 0 // Free tier
  },
  {
    name: 'Startup - Overage',
    segment: 'startup', 
    usage: 5000,
    pricing: startupPricing,
    expected: 80 // 4,000 × $0.02
  },
  {
    name: 'Growth - Within Plan',
    segment: 'growth',
    usage: 8000,
    pricing: growthPricing,
    expected: 0 // Included in plan
  },
  {
    name: 'Growth - With Overage',
    segment: 'growth',
    usage: 25000,
    pricing: growthPricing,
    expected: 150 // 15,000 × $0.01
  },
  {
    name: 'Enterprise - High Volume',
    segment: 'enterprise',
    usage: 750000,
    pricing: enterprisePricing,
    expected: 3000 // 750,000 × $0.004
  }
];

// Run scenarios
scenarios.forEach(scenario => {
  const result = simulator.simulate({
    customerId: `${scenario.segment}_customer`,
    periodStart: '2024-01-01',
    periodEnd: '2024-02-01',
    usageItems: [{
      metric: 'api_calls',
      quantity: scenario.usage,
      priceConfig: scenario.pricing
    }]
  });
  
  console.log(`${scenario.name}:`);
  console.log(`  Usage: ${scenario.usage.toLocaleString()} API calls`);
  console.log(`  Bill: $${result.total}`);
  console.log(`  Expected: $${scenario.expected}`);
  console.log(`  Match: ${result.total === scenario.expected ? '✅' : '❌'}\n`);
});
```

## Scenario 2: Data Processing Platform - DataFlow

**Business Model:** ETL/data processing service with volume-based pricing and committed spend discounts.

### Pricing Structure

```typescript
const dataProcessingPricing = {
  model: 'volume',
  currency: 'USD',
  tiers: [
    { upTo: 100, unitPrice: 0.50 },     // $0.50 per GB (small jobs)
    { upTo: 1000, unitPrice: 0.35 },    // $0.35 per GB (medium jobs)  
    { upTo: 10000, unitPrice: 0.25 },   // $0.25 per GB (large jobs)
    { upTo: null, unitPrice: 0.15 }     // $0.15 per GB (enterprise)
  ]
};

const storageConfig = {
  model: 'flat',
  currency: 'USD',
  unitPrice: 0.025, // $0.025 per GB-hour
  minimumCharge: 1.00 // Minimum $1/month per customer
};

// Enterprise customer with commitment
const enterpriseCommitment = {
  amount: 5000, // $5,000 committed spend
  startDate: '2024-01-01',
  endDate: '2024-12-31', 
  applied: 2500 // $2,500 already used
};
```

### Multi-Metric Scenario

```typescript
const dataFlowScenario = {
  customerId: 'enterprise_dataflow',
  periodStart: '2024-01-01', 
  periodEnd: '2024-02-01',
  usageItems: [
    {
      metric: 'data_processed_gb',
      quantity: 15000, // 15 TB processed
      priceConfig: dataProcessingPricing
    },
    {
      metric: 'storage_gb_hours', 
      quantity: 744000, // 1 TB stored for full month (744 hours)
      priceConfig: storageConfig
    }
  ],
  commitments: [enterpriseCommitment],
  credits: [
    { amount: 500, reason: 'Migration assistance credit' }
  ]
};

const invoice = simulator.simulate(dataFlowScenario);

console.log('DataFlow Enterprise Customer - January 2024');
console.log('===========================================');
invoice.lineItems.forEach(item => {
  console.log(`${item.metric}: ${item.quantity.toLocaleString()} units × $${item.unitPrice} = $${item.subtotal}`);
});
console.log(`Subtotal: $${invoice.subtotal}`);
console.log(`Commitment Applied: -$${invoice.credits}`);  
console.log(`Final Total: $${invoice.total}`);
```

## Scenario 3: Storage Provider - CloudStore

**Business Model:** Cloud storage with graduated pricing (base fees + usage) and retention tiers.

### Pricing Configuration

```typescript
const storagePricing = {
  hot_storage: {
    model: 'graduated',
    currency: 'USD',
    tiers: [
      { upTo: 100, flatPrice: 5, unitPrice: 0.023 },    // $5 base + $0.023/GB
      { upTo: 1000, flatPrice: 15, unitPrice: 0.021 },  // $15 base + $0.021/GB  
      { upTo: null, flatPrice: 50, unitPrice: 0.018 }   // $50 base + $0.018/GB
    ]
  },
  cold_storage: {
    model: 'graduated',
    currency: 'USD', 
    tiers: [
      { upTo: 1000, flatPrice: 2, unitPrice: 0.008 },   // $2 base + $0.008/GB
      { upTo: 10000, flatPrice: 8, unitPrice: 0.006 },  // $8 base + $0.006/GB
      { upTo: null, flatPrice: 25, unitPrice: 0.004 }   // $25 base + $0.004/GB
    ]
  },
  archive_storage: {
    model: 'flat',
    currency: 'USD',
    unitPrice: 0.001 // $0.001/GB - cheap but slow retrieval
  },
  data_retrieval: {
    model: 'tiered',
    currency: 'USD',
    tiers: [
      { upTo: 10, unitPrice: 0.00 },      // 10 GB free retrieval
      { upTo: 100, unitPrice: 0.05 },     // $0.05/GB 
      { upTo: null, unitPrice: 0.03 }     // $0.03/GB bulk
    ]
  }
};
```

### Customer Usage Patterns

```typescript
const storageScenarios = [
  {
    name: 'Small Business',
    usage: {
      hot_storage: 50,      // 50 GB hot storage
      cold_storage: 200,    // 200 GB cold storage  
      archive_storage: 0,   // No archive
      data_retrieval: 5     // 5 GB retrieval (free)
    }
  },
  {
    name: 'Growing Startup',
    usage: {
      hot_storage: 500,     // 500 GB hot
      cold_storage: 2000,   // 2 TB cold
      archive_storage: 5000, // 5 TB archive
      data_retrieval: 50    // 50 GB retrieval
    }
  },
  {
    name: 'Enterprise',
    usage: {
      hot_storage: 5000,    // 5 TB hot
      cold_storage: 25000,  // 25 TB cold
      archive_storage: 100000, // 100 TB archive
      data_retrieval: 200   // 200 GB retrieval
    }
  }
];

// Calculate bills for each scenario
storageScenarios.forEach(scenario => {
  const usageItems = Object.entries(scenario.usage).map(([metric, quantity]) => ({
    metric,
    quantity,
    priceConfig: storagePricing[metric]
  }));
  
  const invoice = simulator.simulate({
    customerId: scenario.name.toLowerCase().replace(' ', '_'),
    periodStart: '2024-01-01',
    periodEnd: '2024-02-01', 
    usageItems
  });
  
  console.log(`${scenario.name} Storage Bill:`);
  console.log(`Total: $${invoice.total}`);
  
  // Show breakdown
  invoice.lineItems.forEach(item => {
    const usage = scenario.usage[item.metric];
    console.log(`  ${item.metric}: ${usage} GB = $${item.subtotal}`);
  });
  console.log('');
});
```

## Scenario 4: Enterprise Platform - WorkflowCloud  

**Business Model:** Complex enterprise platform with multiple pricing dimensions and custom contracts.

### Multi-Dimensional Pricing

```typescript
const workflowPricing = {
  compute_hours: {
    model: 'tiered',
    currency: 'USD',
    tiers: [
      { upTo: 100, unitPrice: 0.50 },    // Dev/test workloads
      { upTo: 1000, unitPrice: 0.35 },   // Production workloads
      { upTo: null, unitPrice: 0.25 }    // Enterprise scale
    ]
  },
  active_workflows: {
    model: 'graduated', 
    currency: 'USD',
    tiers: [
      { upTo: 10, flatPrice: 0, unitPrice: 0 },        // Free tier
      { upTo: 100, flatPrice: 50, unitPrice: 2.00 },   // Professional
      { upTo: null, flatPrice: 200, unitPrice: 1.50 }  // Enterprise
    ]
  },
  api_requests: {
    model: 'volume',
    currency: 'USD', 
    tiers: [
      { upTo: 100000, unitPrice: 0.0001 },   // $0.10 per 1K requests
      { upTo: 1000000, unitPrice: 0.00008 }, // Volume discount
      { upTo: null, unitPrice: 0.00005 }     // Enterprise rate
    ]
  },
  data_transfer_gb: {
    model: 'tiered',
    currency: 'USD',
    tiers: [
      { upTo: 100, unitPrice: 0.09 },     // First 100 GB
      { upTo: 1000, unitPrice: 0.085 },   // Next 900 GB  
      { upTo: null, unitPrice: 0.08 }     // Above 1 TB
    ]
  }
};
```

### Enterprise Contract Simulation

```typescript
const enterpriseContract = {
  customerId: 'acme_corp',
  periodStart: '2024-01-01',
  periodEnd: '2024-02-01',
  usageItems: [
    {
      metric: 'compute_hours',
      quantity: 5000,
      priceConfig: workflowPricing.compute_hours
    },
    {
      metric: 'active_workflows', 
      quantity: 250,
      priceConfig: workflowPricing.active_workflows
    },
    {
      metric: 'api_requests',
      quantity: 2500000,
      priceConfig: workflowPricing.api_requests
    },
    {
      metric: 'data_transfer_gb',
      quantity: 1500,
      priceConfig: workflowPricing.data_transfer_gb
    }
  ],
  commitments: [
    {
      amount: 10000, // $10K annual commitment
      startDate: '2024-01-01',
      endDate: '2024-12-31',
      applied: 0 // First month
    }
  ],
  credits: [
    { amount: 1000, reason: 'Migration support' },
    { amount: 500, reason: 'Q1 promotion' }
  ],
  taxRate: 0 // Enterprise - tax handled separately
};

const enterpriseInvoice = simulator.simulate(enterpriseContract);

console.log('ACME Corp - Enterprise Contract');
console.log('==============================');
console.log(`Period: ${enterpriseContract.periodStart} to ${enterpriseContract.periodEnd}`);
console.log('');

console.log('Usage Breakdown:');
enterpriseInvoice.lineItems.forEach(item => {
  console.log(`${item.metric.replace('_', ' ').toUpperCase()}:`);
  console.log(`  Quantity: ${item.quantity.toLocaleString()}`);
  console.log(`  Rate: $${item.unitPrice}`);
  console.log(`  Subtotal: $${item.subtotal}`);
  console.log('');
});

console.log('Billing Summary:');
console.log(`Subtotal: $${enterpriseInvoice.subtotal}`);
console.log(`Credits Applied: -$${enterpriseInvoice.credits}`);
console.log(`Commitment Applied: $${Math.min(enterpriseInvoice.total, 10000/12).toFixed(2)}`);
console.log(`Total Due: $${enterpriseInvoice.total}`);
```

## Scenario 5: Pricing Optimization Study

**Goal:** Compare different pricing strategies to maximize revenue while maintaining customer satisfaction.

### A/B Testing Different Models

```typescript
const usagePatterns = [
  { segment: 'small', avgUsage: 2500, customers: 1000 },
  { segment: 'medium', avgUsage: 15000, customers: 500 },
  { segment: 'large', avgUsage: 75000, customers: 100 },
  { segment: 'enterprise', avgUsage: 300000, customers: 20 }
];

const pricingOptions = {
  current: {
    model: 'tiered',
    tiers: [
      { upTo: 5000, unitPrice: 0.01 },
      { upTo: 50000, unitPrice: 0.008 },
      { upTo: null, unitPrice: 0.005 }
    ]
  },
  option_a: {
    model: 'volume', 
    tiers: [
      { upTo: 5000, unitPrice: 0.012 },
      { upTo: 50000, unitPrice: 0.009 },
      { upTo: null, unitPrice: 0.006 }
    ]
  },
  option_b: {
    model: 'graduated',
    tiers: [
      { upTo: 10000, flatPrice: 25, unitPrice: 0.005 },
      { upTo: 100000, flatPrice: 100, unitPrice: 0.003 },
      { upTo: null, flatPrice: 400, unitPrice: 0.002 }
    ]
  }
};

// Revenue analysis
console.log('Pricing Strategy Analysis');
console.log('========================');

Object.entries(pricingOptions).forEach(([strategyName, pricing]) => {
  let totalRevenue = 0;
  
  console.log(`\n${strategyName.toUpperCase()} Strategy:`);
  
  usagePatterns.forEach(pattern => {
    const invoice = simulator.simulate({
      customerId: pattern.segment,
      periodStart: '2024-01-01',
      periodEnd: '2024-02-01',
      usageItems: [{
        metric: 'api_calls',
        quantity: pattern.avgUsage,
        priceConfig: { model: pricing.model, currency: 'USD', tiers: pricing.tiers }
      }]
    });
    
    const segmentRevenue = invoice.total * pattern.customers;
    totalRevenue += segmentRevenue;
    
    console.log(`  ${pattern.segment}: $${invoice.total} × ${pattern.customers} = $${segmentRevenue.toLocaleString()}`);
  });
  
  console.log(`  TOTAL MONTHLY REVENUE: $${totalRevenue.toLocaleString()}`);
});
```

## Testing Best Practices

### 1. Edge Case Testing

```typescript
const edgeCases = [
  { name: 'Zero usage', quantity: 0 },
  { name: 'Exact tier boundary', quantity: 1000 },
  { name: 'Just over boundary', quantity: 1001 },
  { name: 'Fractional usage', quantity: 999.99 },
  { name: 'Very high usage', quantity: 10000000 }
];

edgeCases.forEach(testCase => {
  const result = simulator.simulate({
    customerId: 'test',
    periodStart: '2024-01-01',
    periodEnd: '2024-02-01',
    usageItems: [{
      metric: 'api_calls',
      quantity: testCase.quantity,
      priceConfig: tieredPricing
    }]
  });
  console.log(`${testCase.name}: $${result.total}`);
});
```

### 2. Consistency Validation

```typescript
// Ensure pricing is monotonic (more usage = higher or equal cost)
const quantities = [0, 100, 1000, 5000, 10000, 50000, 100000];

quantities.forEach((quantity, index) => {
  if (index === 0) return;
  
  const current = simulator.simulate({
    customerId: 'test',
    periodStart: '2024-01-01', 
    periodEnd: '2024-02-01',
    usageItems: [{ metric: 'test', quantity, priceConfig: tieredPricing }]
  });
  
  const previous = simulator.simulate({
    customerId: 'test',
    periodStart: '2024-01-01',
    periodEnd: '2024-02-01', 
    usageItems: [{ metric: 'test', quantity: quantities[index-1], priceConfig: tieredPricing }]
  });
  
  if (current.total < previous.total) {
    console.error(`❌ Pricing inconsistency: ${quantity} usage costs less than ${quantities[index-1]}`);
  }
});
```

## Integration with CI/CD

```typescript
// Example test for automated pricing validation
describe('Pricing Regression Tests', () => {
  test('Known customer scenarios should maintain expected costs', () => {
    const knownScenarios = [
      { usage: 1000, expectedCost: 10.00 },
      { usage: 5000, expectedCost: 42.00 },
      { usage: 20000, expectedCost: 146.00 }
    ];
    
    knownScenarios.forEach(scenario => {
      const result = simulator.simulate({
        customerId: 'regression_test',
        periodStart: '2024-01-01',
        periodEnd: '2024-02-01',
        usageItems: [{
          metric: 'api_calls',
          quantity: scenario.usage,
          priceConfig: productionPricing
        }]
      });
      
      expect(result.total).toBeCloseTo(scenario.expectedCost, 2);
    });
  });
});
```

These scenarios demonstrate how the StripeMeter pricing simulator can be used to test, validate, and optimize pricing strategies across different business models and customer segments.


path: docs/welcome.md
content: # Welcome to StripeMeter

StripeMeter helps you **prove and enforce pre-invoice parity** for Stripe usage billing.

---

## 1) Ingest (send usage events, exactly once)
- Endpoint: `/v1/events/ingest`  
- Send usage with an **Idempotency-Key** so duplicates are counted once.  
- Try it: see the 30-sec demo in the README.

**Read next:** [Ingest API](./api/ingest.md)

---

## 2) Replay (fix late events within a watermark window)
- Endpoint: `POST /v1/replay`  
- Run **dry-run** to see the delta, or **apply** to update counters.  
- Deterministic **parity report** via the Reconciliation Summary.

**Read next:** [Reconciliation Summary API](./api/reconciliation-summary.md)

---

## 3) Reconcile (prove drift → 0 before finalization)
- Endpoint: `GET /v1/reconciliation/summary`  
- View **drift** and **counters before/after**.  
- Use the **Parity Visualizer** to see the time-series change after replay.

**Read next:** [Reconciliation Summary API](./api/reconciliation-summary.md)

---

## 4) Usage History (visualize counters over time)
- Endpoint: `GET /v1/usage/history`  
- Returns **time-series** and **totals** for dashboards before/after replay.

**Read next:** [Usage History API](./api/usage-history.md)


path: ops/ALERTS.md
content: ## Alerting Rules (Prometheus)

This document defines baseline alerting rules for StripeMeter. Each alert includes a brief next action.

Copy the rules block into your Prometheus `rule_files` and run `promtool check rules` before applying.

```yaml
groups:
  - name: stripemeter.rules
    rules:
      # Drift absolute exceeds epsilon for 15 minutes
      - alert: ReconciliationDriftAbsHigh
        expr: max_over_time(reconciliation_diff_abs[15m]) > 0.0
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Reconciliation absolute drift above epsilon"
          description: "Absolute drift between local and Stripe exceeds epsilon for 15m. value={{ $value }}"
          runbook: "Inspect reconciliation reports; follow RECONCILIATION.md steps 1–4."

      # Drift percentage exceeds threshold for 15 minutes
      - alert: ReconciliationDriftPctHigh
        expr: max_over_time(reconciliation_diff_pct[15m]) > 0.005
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "Reconciliation percentage drift above threshold"
          description: "Percentage drift (|local - stripe| / stripe) > 0.5% for 15m. value={{ $value }}"
          runbook: "Backfill or adjust per RECONCILIATION.md for affected item(s)."

      # Queue lag above threshold (requires exporter of BullMQ/Redis lag)
      - alert: IngestionQueueLagHigh
        expr: max_over_time(queue_lag_seconds{queue=~"aggregation|writer"}[15m]) > 60
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Queue lag over 60s"
          description: "Background queue lag above 60s for 15m. value={{ $value }}"
          runbook: "Check Redis, worker replicas, backlog; scale workers or unstick jobs."

      # Ingest latency p95 above SLO
      - alert: IngestLatencyHighP95
        expr: histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{route="/v1/events/ingest"}[5m]))) > 0.2
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Ingest p95 latency over 200ms"
          description: "POST /v1/events/ingest p95 > 200ms for 15m. value={{ $value }}"
          runbook: "Check API/DB/Redis; scale API or tune batch sizes."

      # HTTP 5xx rate above threshold
      - alert: Http5xxRateHigh
        expr: sum by (route) (rate(http_requests_total{status_code=~"5.."}[5m])) > 0.1
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "HTTP 5xx rate high"
          description: "5xx error rate > 0.1 req/s over 5m for 15m. route={{ $labels.route }} value={{ $value }}"
          runbook: "Check deploys/logs/upstreams; roll back or hotfix."
```

Notes
- Metrics used are based on built-in exporters:
  - `http_request_duration_seconds_bucket`, `http_requests_total` from API
  - `reconciliation_diff_abs`, `reconciliation_diff_pct` are placeholders; wire to reconciler metrics or Redis-exported gauges.
  - `queue_lag_seconds` is a placeholder for BullMQ/Redis consumer lag. Add an exporter or custom gauge.
- Coordinate thresholds with `RECONCILIATION.md` and production SLOs in `project-principles.md`.

### Alertmanager Example

See `ops/alertmanager/alertmanager.yml` for a minimal route/receiver.


### What to do next (Runbook)

When any reconciliation alert fires, follow the operator playbook:

- Read `RECONCILIATION.md` → Steps 1–6 (verify health, inspect drift, replay window, confirm resolution)
- Use copy-paste commands for `/metrics`, Replay API, and reconciliation summary
- Validate drift is back within epsilon (≤ 0.5%) before closing the alert




Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)