npm package: @mondaydotcomorg/monday-api-queue-sdk
Description: A module for executing / consuming monday.com API requests at scale.
Author: monday.com SE
Latest version: 0.1.14

# monday-api-queue-sdk

A module for executing and consuming monday.com API requests at scale using queue-based processing.

## Overview

This SDK provides a robust solution for handling high-volume monday.com API requests through a queue-based architecture. It handles API rate limiting, automatic retries with exponential backoff, and callback notifications when requests complete.

### Key Features

- Queue-based processing for scalable API request handling
- Built-in retry logic with exponential backoff
- Automatic handling of monday.com rate limits (complexity budget exhaustion, concurrency limits)
- Support for AWS SQS as the message queue
- S3 integration for storing API response payloads
- Callback notifications via HTTP endpoints
- Both long-polling consumers and trigger-based (Lambda) consumers

## Installation

```bash
npm install @mondaydotcomorg/monday-api-queue-sdk
```

## Architecture

The SDK uses a two-queue architecture:

```
┌─────────────┐     ┌─────────────────────┐     ┌───────────────┐
│  API Queue  │────▶│  ApiRequestProcessor │────▶│ Callback Queue│
└─────────────┘     └─────────────────────┘     └───────────────┘
                              │                         │
                              ▼                         ▼
                         ┌─────────┐         ┌──────────────────────┐
                         │   S3    │         │ CallbackQueueProcessor│
                         └─────────┘         └──────────────────────┘
                                                        │
                                                        ▼
                                                ┌──────────────┐
                                                │ Your Webhook │
                                                └──────────────┘
```

1. **API Queue**: Receives monday.com API requests
2. **ApiRequestQueueProcessor**: Executes the API call, uploads response to S3, pushes to callback queue
3. **Callback Queue**: Holds completed requests ready for delivery
4. **CallbackQueueProcessor**: Delivers results to your webhook endpoint

## Quick Start

### Using the Factory (Recommended)

The `QueueFactory` provides a convenient way to create pre-configured processors. It internally creates the necessary producers for retries and pushing results to the callback queue:

```typescript
import {
  QueueFactory,
  QueueConfigTypes,
  FileStorageTypes,
  TriggerConsumer,
} from "@mondaydotcomorg/monday-api-queue-sdk";

const apiProcessor = QueueFactory.createApiProcessor({
  apiQueueSettings: {
    type: QueueConfigTypes.SQS,
    queueUrl: "https://sqs.us-east-1.amazonaws.com/123456789/api-queue",
    region: "us-east-1",
  },
  callbackQueueSettings: {
    type: QueueConfigTypes.SQS,
    queueUrl: "https://sqs.us-east-1.amazonaws.com/123456789/callback-queue",
    region: "us-east-1",
  },
  fileStorageSettings: {
    type: FileStorageTypes.S3,
    bucketName: "my-api-responses-bucket",
  },
  maxRetries: 3,
  shouldRetryOnGeneralError: true,
});

const callbackProcessor = QueueFactory.createCallbackProcessor({
  queueSettings: {
    type: QueueConfigTypes.SQS,
    queueUrl: "https://sqs.us-east-1.amazonaws.com/123456789/callback-queue",
    region: "us-east-1",
  },
  maxRetries: 3,
  shouldRetryOnGeneralError: true,
});
```

### Lambda / Trigger-Based Consumer

For AWS Lambda or other serverless environments:

```typescript
import {
  TriggerConsumer,
  parseSqsRecords,
  ApiRequestQueueJob,
} from "@mondaydotcomorg/monday-api-queue-sdk";

export const handler = async (event: SQSEvent) => {
  const consumer = new TriggerConsumer(apiProcessor);
  const jobs = parseSqsRecords<ApiRequestQueueJob>(event.Records);
  
  const result = await consumer.consumeBatch(jobs);
  
  console.log(`Processed: ${result.successful.length} successful, ${result.failed.length} failed`);
};
```

### Enqueueing API Requests

The `QueueFactory` creates producers internally for retries and callback delivery, but you'll need to create your own `SqsProducer` to initially enqueue jobs to the API queue:

```typescript
import { SqsProducer, ApiRequestQueueJob } from "@mondaydotcomorg/monday-api-queue-sdk";

const producer = new SqsProducer({
  queueUrl: "https://sqs.us-east-1.amazonaws.com/123456789/api-queue",
  region: "us-east-1",
});

const job: ApiRequestQueueJob = {
  token: "your-monday-api-token",
  query: `query { boards(limit: 10) { id name } }`,
  variables: {},
  data: {},
};

const result = await producer.enqueue(job);
if (result.success) {
  console.log("Job enqueued with ID:", result.id);
}
```

## API Reference

### Queue Jobs

#### ApiRequestQueueJob

Job structure for API requests:

```typescript
interface ApiRequestQueueJob {
  token: string;
  query: string;
  variables?: Record<string, unknown>;
  id?: string | number | null;
  attempts?: number;
  data: unknown;
}
```

#### CallbackQueueJob

Job structure for callbacks:

```typescript
interface CallbackQueueJob {
  callbackUrl: string;
  payload: unknown;
  id?: string | number | null;
  attempts?: number;
  data: unknown;
}
```

### Processors

#### ApiRequestQueueProcessor

Processes monday.com API requests:

- Executes GraphQL queries using the provided token
- Handles rate limiting (COMPLEXITY_BUDGET_EXHAUSTED, MAX_CONCURRENCY_EXCEEDED)
- Uploads responses to S3
- Pushes results to the callback queue
- Automatic retry with exponential backoff

#### CallbackQueueProcessor

Delivers results to webhook endpoints:

- Makes HTTP POST requests to callback URLs
- Retries on 5xx errors
- Configurable retry behavior

### Consumers

#### TriggerConsumer

For serverless/Lambda environments:

```typescript
const consumer = new TriggerConsumer(processor);
const result = await consumer.consumeBatch(jobs);
```

#### SqsConsumer

Abstract base class for long-polling SQS consumers. Extend this for continuous processing:

```typescript
class MyConsumer extends SqsConsumer {
  constructor() {
    super(processor, {
      queueUrl: "https://sqs...",
      region: "us-east-1",
    });
  }
}

const consumer = new MyConsumer();
await consumer.start();
```

### Producers

#### SqsProducer

Enqueue jobs to SQS:

```typescript
const producer = new SqsProducer({
  queueUrl: "https://sqs.us-east-1.amazonaws.com/123456789/queue",
  region: "us-east-1",
});

await producer.enqueue(job, { delay: 30 });
```

### Utilities

#### parseSqsRecords

Parse SQS event records into typed jobs:

```typescript
import { parseSqsRecords, ApiRequestQueueJob } from "@mondaydotcomorg/monday-api-queue-sdk";

const jobs = parseSqsRecords<ApiRequestQueueJob>(event.Records);
```

## Configuration

### Queue Settings

```typescript
interface SQSVendorQueueConfig {
  type: QueueConfigTypes.SQS;
  queueUrl: string;
  region: string;
}
```

### File Storage Settings

```typescript
interface S3FileStorageConfig {
  type: FileStorageTypes.S3;
  bucketName: string;
}
```

### Processor Options

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `maxRetries` | number | 3 | Maximum retry attempts |
| `shouldRetryOnGeneralError` | boolean | true | Retry on non-monday.com errors |

## Error Handling

The SDK includes specialized error types:

### MondayError

Thrown when the monday.com API returns errors:

```typescript
import { MondayError, MondayErrorCodes } from "@mondaydotcomorg/monday-api-queue-sdk";

if (error instanceof MondayError) {
  const hasRateLimit = error.mondayErrors.some(
    (e) => e.extensions?.code === MondayErrorCodes.COMPLEXITY_BUDGET_EXHAUSTED
  );
}
```

### JobError

Thrown during job processing with retry information:

```typescript
import { JobError } from "@mondaydotcomorg/monday-api-queue-sdk";

if (error instanceof JobError) {
  console.log("Should retry:", error.shouldRetry);
  console.log("Retry delay:", error.options?.delay);
}
```

## AWS Requirements

### IAM Permissions

The SDK requires the following AWS permissions:

**SQS:**
- `sqs:SendMessage`
- `sqs:ReceiveMessage`
- `sqs:DeleteMessage`
- `sqs:GetQueueAttributes`

**S3:**
- `s3:PutObject`

### Queue Configuration

Recommended SQS queue settings:
- Visibility Timeout: 60 seconds (adjust based on API response times)
- Message Retention: 4 days
- Dead Letter Queue: Recommended for failed messages

## Development

```bash
npm install

npm run build

npm run test

npm run test:coverage

npm run lint

npm run typecheck
```

## License

MIT