Logstash — Dead Letter Queue. Logstash is an open-source, server-side…
By: Vishnu G
Tags: Logstash, Elasticsearch, Elk Stack

Logstash is an open-source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously transforms it, and then sends it to your favourite “stash” like Elasticsearch.

The data collection is accomplished via configurable input plugins including raw socket/packet communication, file tailing, and several message bus clients. Once an input plugin has collected data it can be processed by any number of filters which modify and annotate the event data. Finally Logstash routes events to output plugins which can forward the events to a variety of external programs including Elasticsearch, local files and several message bus implementations.

One of the main problems the users face when Logstash encounters an event that it cannot process because the data contains a mapping error or some other issue. In that case, Logstash pipeline either hangs or drops the unsuccessful event. In order to protect against data loss in this situation, we can configure Logstash to write unsuccessful events to a dead letter queue instead of dropping them.

To process the events in the dead letter queue, we need to create a Logstash pipeline configuration using the dead letter queue. input plugin.

The dead letter queue feature is currently supported for the Elasticsearch output plugin output only. Additionally, the dead letter queue is only used where the response code is either 400 or 404, both of which indicate an event that cannot be retried. Support for additional outputs will be available in future releases of the Logstash plugins.

Configuring Dead Letter Queue

Dead letter queues are disabled by default. To enable dead letter queues, set the dead_letter_queue_enable option in the logstash.yml.

We can also specify the path for the queue.

By default it is /logstash-<version>/data/dead_letter_queue

We can check the status of the queue by calling the logstash stats API.

Processing the contents of Dead Letter Queue

We can use Logstash to reprocess the content of a dead letter queue by using the dead_lette_queue plugin

Details of Dead_letter_queue plugin

Example file

Example Use Case

Below is a use case for the use of the dead letter queue.

Load a JSON file to Elasticsearch using logstash

{“fruit”: “Apple”,”size”: “Large”,”color”: “Red”,”id”: 1,”isFruit”: true} {“fruit”: “Apple”,”size”: “Large”,”color”: “Red”,”id”: 2,”isFruit”: true} {“fruit”: “Apple”,”size”: “Large”,”color”: “Red”,”id”: 3,”isFruit”: true} {“fruit”: “Apple”,”size”: “Large”,”color”: “Red”,”id”: 4,”isFruit”: false} {“fruit”: “Apple”,”size”: “Large”,”color”: “Red”,”id”: 5,”isFruit”: false}

Logstash pipeline config

Failure case

Try to index a dataset that has a different field type

{“fruit”: “Apple”,”size”: “Large”,”color”: “Red”,”id”: 5,”isFruit”: 56} {“fruit”: “Apple”,”size”: “Large”,”color”: “Red”,”id”: 5,”isFruit”: 56} {“fruit”: “Apple”,”size”: “Large”,”color”: “Red”,”id”: 5,”isFruit”: 56} {“fruit”: “Apple”,”size”: “Large”,”color”: “Red”,”id”: 5,”isFruit”: 56} {“fruit”: “Apple”,”size”: “Large”,”color”: “Red”,”id”: 5,”isFruit”: 56} {“fruit”: “Apple”,”size”: “Large”,”color”: “Red”,”id”: true,”isFruit”: 56} {“fruit”: “Apple”,”size”: “Large”,”color”: “Red”,”id”: true,”isFruit”: 56} {“fruit”: “Apple”,”size”: “Large”,”color”: “Red”,”id”: true,”isFruit”: 56}

Here the type of id and isFruit is wrong. When we try to index this data set, elasticsearch will through MapperParserException.

In this case Logstash stores the event to dead letter queue. We can check this by calling the stats API of Logstash.

Queue details

Reading data from the dead letter queue

This pipeline will read the data from queue and index to elasticsearch.

Reading the data from the queue will not delete the event from the queue. This functionality is still in development(https://github.com/elastic/logstash/issues/8904)