npm package: @sygnl/dedupe
Description: In-memory event deduplication with time-window tracking for edge workers and high-throughput systems
Author: Edge Foundry, Inc.
Latest version: 1.0.0

# @sygnl/dedupe

> In-memory event deduplication with time-window tracking for edge workers and high-throughput systems.

## Install

```bash
npm install @sygnl/dedupe
```

## Quick Start

```typescript
import { DedupeManager } from '@sygnl/dedupe';

const dedupe = new DedupeManager({
  configs: {
    purchase: {
      enabled: true,
      dedupeKey: 'order.id',
      windowMs: 300000 // 5 minutes
    },
    add_to_cart: {
      enabled: true,
      dedupeKey: 'cart.id',
      windowMs: 60000 // 1 minute
    }
  }
});

// Check for duplicates
const result = dedupe.check('purchase', event, 'evt_123', 'webhook');

if (result.isDuplicate) {
  console.log('Duplicate detected!');
  console.log('Previous event:', result.previousEventId);
  console.log('Age:', result.ageMs, 'ms');
  return; // Skip processing
}

// Process event...
```

## Use Cases

- **Prevent double-clicks** - Stop users from submitting forms multiple times
- **Webhook retries** - Avoid processing the same webhook twice
- **Multiple sources** - Dedupe events from pixel + webhook for same order
- **Edge workers** - No external storage needed (KV, Redis)
- **High throughput** - In-memory, sub-millisecond lookups

## How It Works

1. **Extract key** from event (e.g., `order.id`, `user.email`)
2. **Check cache** if key seen within time window
3. **Return duplicate** if found, or **record as new**
4. **Auto-cleanup** expired entries to prevent memory leaks

## API

### `DedupeManager`

```typescript
const dedupe = new DedupeManager({
  configs?: Record<string, DedupeConfig>,
  defaultWindowMs?: number,      // Default: 60000 (1 min)
  cleanupIntervalMs?: number,    // Default: 60000 (1 min)
  maxAgeMs?: number,             // Default: 600000 (10 min)
  debug?: boolean                // Default: false
});
```

### `check(eventType, event, eventId, source?, metadata?): DedupeResult`

Check if event is a duplicate.

```typescript
const result = dedupe.check('purchase', {
  order: { id: 'order_123' },
  total: 99.99
}, 'evt_abc', 'webhook');

// Result
{
  isDuplicate: false,
  dedupeKey: 'order_123',
  windowMs: 300000
}
```

### `configure(eventType, config): void`

Configure dedupe for an event type at runtime.

```typescript
dedupe.configure('subscription_created', {
  enabled: true,
  dedupeKey: 'subscription.id',
  windowMs: 120000
});
```

## Dedupe Key Extraction

### Object paths

```typescript
// Nested object
{ dedupeKey: 'user.profile.email' }
// Event: { user: { profile: { email: 'test@example.com' } } }
// Key: 'test@example.com'
```

### Array indexing

```typescript
// First product ID
{ dedupeKey: 'products.0.id' }
// Event: { products: [{ id: 'prod_123' }, { id: 'prod_456' }] }
// Key: 'prod_123'
```

### Custom extractor

```typescript
{
  dedupeKey: (event) => {
    // Custom logic
    return event.items.map(i => i.id).join(',');
  }
}
// Event: { items: [{ id: 'A' }, { id: 'B' }] }
// Key: 'A,B'
```

## Examples

### E-commerce

```typescript
const dedupe = new DedupeManager({
  configs: {
    purchase: {
      enabled: true,
      dedupeKey: 'order.id',
      windowMs: 300000 // 5 min - prevent double submissions
    },
    add_to_cart: {
      enabled: true,
      dedupeKey: (e) => `${e.user.id}:${e.product.id}`,
      windowMs: 60000 // 1 min - same user adding same product
    }
  }
});
```

### Webhook Processing

```typescript
// Cloudflare Worker
export default {
  async fetch(request, env) {
    const event = await request.json();
    
    const result = dedupe.check(
      event.type,
      event,
      event.id,
      'webhook'
    );
    
    if (result.isDuplicate) {
      return new Response('Duplicate', { status: 200 });
    }
    
    await processEvent(event);
    return new Response('OK');
  }
};
```

### SaaS Events

```typescript
const dedupe = new DedupeManager({
  configs: {
    user_signup: {
      enabled: true,
      dedupeKey: 'user.email',
      windowMs: 120000 // 2 min
    },
    subscription_started: {
      enabled: true,
      dedupeKey: 'subscription.id',
      windowMs: 600000 // 10 min
    }
  }
});
```

## Utility Methods

```typescript
// Clear all state (testing)
dedupe.clear();

// Get count of tracked events
const count = dedupe.getCount(); // 42

// Get keys for event type
const keys = dedupe.getKeysForType('purchase'); // ['order_123', 'order_456']

// Get specific entry
const entry = dedupe.getEntry('purchase', 'order_123');
// { eventId: 'evt_abc', timestamp: 1234567890, source: 'webhook' }

// Force cleanup
const removed = dedupe.forceCleanup(); // 15
```

## Configuration Options

### Per-Event Config

```typescript
interface DedupeConfig {
  enabled: boolean;
  dedupeKey: string | ((event: any) => string | null);
  windowMs: number;
}
```

### Global Options

```typescript
interface DedupeManagerOptions {
  configs?: Record<string, DedupeConfig>;
  defaultWindowMs?: number;       // Default window for unconfigured events
  cleanupIntervalMs?: number;     // How often to cleanup expired entries
  maxAgeMs?: number;              // Max age before entry is removed
  debug?: boolean;                // Enable debug logging
}
```

## Time Windows

Common window durations:

- **1 second**: `1000`
- **30 seconds**: `30000`
- **1 minute**: `60000`
- **5 minutes**: `300000`
- **10 minutes**: `600000`
- **1 hour**: `3600000`

## Performance

- **Lookup**: O(1) - Hash map lookup
- **Cleanup**: O(n) - Runs every `cleanupIntervalMs`
- **Memory**: ~200 bytes per tracked event

## Edge Cases

- **Missing keys**: Pass through (not duplicate)
- **Null events**: Pass through
- **No config**: Pass through
- **Disabled**: Pass through
- **Expired**: Treated as new event

## License

Apache-2.0