npm package: @shutterstock/kinesis-helpers
Description: Helper classes for `@aws-sdk/client-kinesis`
Latest version: 1.0.15

[![npm (scoped)](https://img.shields.io/npm/v/%40shutterstock/kinesis-helpers)](https://www.npmjs.com/package/@shutterstock/kinesis-helpers) [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT) [![API Docs](https://img.shields.io/badge/API%20Docs-View%20Here-blue)](https://tech.shutterstock.com/kinesis-helpers/) [![Build - CI](https://github.com/shutterstock/kinesis-helpers/actions/workflows/ci.yml/badge.svg)](https://github.com/shutterstock/kinesis-helpers/actions/workflows/ci.yml) [![Package and Publish](https://github.com/shutterstock/kinesis-helpers/actions/workflows/publish.yml/badge.svg)](https://github.com/shutterstock/kinesis-helpers/actions/workflows/publish.yml) [![Publish Docs](https://github.com/shutterstock/kinesis-helpers/actions/workflows/docs.yml/badge.svg)](https://github.com/shutterstock/kinesis-helpers/actions/workflows/docs.yml)

# Overview

Helper classes for the AWS v3 SDK for JS Kinesis Client (`@aws-sdk/client-kinesis`).

`KinesisRetrier` will retry failed records with a "successful" `PutRecordsCommand` that has items marked as failed within the response, which then need to be retried.

`KinesisBackgroundWriter` will accept writes immediately if there is a concurrent writer slot available, and will block the caller until a slot becomes available if all slots are in use.

# Getting Started

## Installation

The package is available on npm as [@shutterstock/kinesis-helpers](https://www.npmjs.com/package/@shutterstock/kinesis-helpers)

`npm i @shutterstock/kinesis-helpers`

## Importing

```typescript
import { KinesisBackgroundWriter, KinesisRetrier } from '@shutterstock/kinesis-helpers';
```

## API Documentation

After installing the package, you might want to look at our [API Documentation](https://tech.shutterstock.com/kinesis-helpers/) to learn about all the features available.

# Features

- [KinesisRetrier](https://tech.shutterstock.com/kinesis-helpers/classes/KinesisRetrier.html)
  - Kinesis batch put retrier
  - Batch puts return a list of items that were throttled
  - The batch is marked as succeeded (200 status code) even though items failed
  - As a result, the built-in AWS SDK retry logic will not retry the items in the batch that were throttle
- [KinesisBackgroundWriter](https://tech.shutterstock.com/kinesis-helpers/classes/KinesisBackgroundWriter.html)
  - Accepts writes immediately if there is a concurrent writer slot available
  - Blocks the caller until a slot becomes available if all slots are in use
  - Collects and exposes errors so the caller can detect if errors have been happening

# Contributing

## Setting up Build Environment

- `nvm use`
- `npm i`
- `npm run build`
- `npm run lint`
- `npm run test`

## Running Examples

### kinesis-retrier

1. Create Kinesis Data Stream using AWS Console or any other method
   1. Example: `aws kinesis create-stream --stream-name kinesis-helpers-test-stream --shard-count 1`
   2. Default name is `kinesis-helpers-test-stream`
   3. 1 shard is sufficient
   4. 1 day retention is sufficient
   5. No encryption is sufficient
   6. On-demand throughput is sufficient
2. `npm run example:kinesis-retrier`
   1. If the stream name was changed: `KINESIS_STREAM_NAME=my-stream-name npm run example:kinesis-retrier`

### kinesis-background-writer

1. Create Kinesis Data Stream using AWS Console or any other method
   1. Example: `aws kinesis create-stream --stream-name kinesis-helpers-test-stream --shard-count 1`
   2. Default name is `kinesis-helpers-test-stream`
   3. 1 shard is sufficient
   4. 1 day retention is sufficient
   5. No encryption is sufficient
   6. On-demand throughput is sufficient
2. `npm run example:kinesis-background-writer`
   1. If the stream name was changed: `KINESIS_STREAM_NAME=my-stream-name npm run example:kinesis-background-writer`