How to properly handle Elasticsearch ingest pipelines failures
By: Benoit Luttringer
Tags: Error Handling, Elasticsearch, Ingest, Pipeline

Handle errors

If the above ingest pipelines fails, that means that one of its processor has failed. That also means that the data received are not like we think they should be (because hey, we already anticipated a lot on the potential issues).

If the netflix-titles-pipeline is left as it is, any parsing error will lead to an indexation error but we will loose the original message, so we have to do something smart to handle the error and store the failed message in a place here we can find it, fix it and replay it : the Dead Letter Index ! 🎉

Create a template

Since Elastic 7.8, index templates are composable. We will use this new way of creating templates to tell Elasticsearch to apply a specified mapping whenever we create an index with a name starting with dead-letter- .

First we create the mapping component template :

And then the index template using this component template :

Let’s analyze the mapping we just created.

Each document stored in an index created with this mapping will contains :

the message that was originally received (and that failed)

the name of the index in which we wanted to add this document at first

the date when the document was created in the dead letter destination

an object error containing the error message, the type and the tag of the processor that failed

This is obviously just an example, feel free to add any field you might need to this mapping.

Create an ingest pipeline to add documents to a dead letter index

This ingest pipeline will be used in case of a failure during the processing of the previous ingest pipeline. It will receive the failed document in the state it was when it failed and that is why the removal of the original message field has to be set at the very last position of the processors list : we need to keep it until the very last moment.

When the failed document will be received through the dead letter ingest pipeline, it will have the message field (containing the raw data) but also some new fields created this far by the ingest pipeline : we will have to clean up the received document to keep only the message field. Then we will have to fill in the other fields of a dead letter document as it has been defined by the mapping we built sooner.

Here is the dead letter ingest pipeline :

Let’s explain this ingest pipeline.

Once again, remember that processors in an ingest pipeline are executed top to bottom.

First processor is the most complex too : it is a script processor — running Painless code— and its function is to clean up the document that has failed at the previous ingestion stage. As we said above, the processor that might have failed is not necessarily the first one and our document can already contains more fields than just the message one we only want to keep. This script cleans up the document by removing all fields except the ones beginning with an _ and the message one.

Then we set the event.index field to the value of the _index field of the failed document : it contains the name of the index in which we were trying to store the document at first.

We also set the date the failed document was ingested through this pipeline, and all the information related to the error.

Finally we set the _index field to the name of the index in which we want to store this document : dead-letter-{{{ _index }}} , meaning that if the original document was about to be stored in the index foo before the ingest error, it will now be stored in the dead-letter-foo index.

Update the ingest pipeline to handle errors

Now that our dead letter ingest pipeline is ready, we have to update the Netflix titles ingest pipeline to make use of it in case of errors.

All we have to do is to add an on_failure statement at the end of the pipeline definition to catch all errors and redirect the failed document to our new pipeline dead-letter-pipeline :

Job done !

Testing Time

Ok, back to our Netflix titles data.

As we saw before the file is a CSV in which each line represents a Netflix title. Some fields are missing or need to be cleaned up, that is why we add the "ignore_missing" : true statement in some of the processors of the ingest pipeline. We also use the "trimming": true option in the csv processor to remove extra whitespaces. Running this ingest pipeline with the Netflix titles CSV file is almost OK : each line is correctly processed and the corresponding document is added in the destination index except for the lines with no date_added field which raise an indexation error.

Introduce some more errors

To be able to test our errors handling we are going to tweak the CSV file to add errors other than the missing date_added field.

For the show with id s2 we remove the first double quote of the cast property (the dedicated processor will fail to parse).

For the show with id s4 we replace the November of the date_added field by Novemb .

For the show with id s6 we replace the release year 2016 by 201G .

Prepare for test

First we delete and recreate the netflix-titles-pipeline and the dead-letter-pipeline : that allows us to start with fresh statistics about their execution.

DELETE _ingest/pipeline/netflix-titles-pipeline PUT _ingest/pipeline/netflix-titles-pipeline

{ ... } DELETE _ingest/pipeline/dead-letter-pipeline PUT _ingest/pipeline/dead-letter-pipeline

{ ... }

Then we delete and recreate the netflix_titles index.

DELETE netflix_titles PUT netflix_titles

{ ... }

At this step we should have empty stats for both the netflix-titles-pipeline and the dead-letter-pipeline .

The command :

GET _nodes/stats/ingest?filter_path=nodes.*.ingest

Should return us something like :

What is important here is the number of call to both pipelines (0).

Ingest data with errors

python es-indexer.py -e localhost:9201 -p netflix-titles-pipeline -s ../netflix_titles_with_errors.csv netflix_titles

Now let’s check the ingest pipelines statistics :

We can see the netflix-titles-pipeline has been called 7787 times which is the number of lines in the CSV file. Besides the dead-letter-pipeline has been called 14 times.

If we check the netflix_titles index we will find 7774 documents in it :

GET netflix_titles/_search

Now if we inspect the content of the dead-letter-netflix_titles index that should have been created we will find that it contains 13 documents. Let’s analyze these documents.

GET dead-letter-netflix_titles/_search

The first hit is

Hurray ! We get our original message in the message field and the details of the error in the other fields (the csv processor failed on a missing quote). We can see that the original targeted index was netflix_titles and we have the creation date of this document in the dead letter index as well.

Now for the others documents in the dead letter index :

The first shows us that the date_added field could not be parsed, the second that the release_year was not an integer and the next ones are all issues about the missing date_added field.

Fix everything

The first part of the job is now done : when a document is sent to Elasticsearch it is correctly indexed if there is no error during its processing by the ingest pipeline, else it is indexed in a dead letter index along with all the information that we need to fix the problem.

From there, next step is obviously to be able to fix and replay the document ingestion, hoping that it will pass.

We could create a CSV file with the corrected lines only and try to index again using the python script but we could also use directly the dead letter index to fix the error and reindex the document. This way we stay in the Elasticsearch ecosystem.

Update failed document

The first step is to find the cause of the failure : the event.error.message gives us a good indication and should allow us to fix the content of the message.

From there, we can update the document directly in the dead letter index.

Let’s work on the s2 show — the one which fails due to a missing quote — and update the message field to add the double quote again.

This command allows us to update the message field but also to add a fix_iteration field to the document that can be useful later : if the fix_iteration field is present, that means someone has tried to fix the document and the value of the fix_iteration indicates how many attempts have been made.

Try again

Now retrying the indexation of the fixed document is as simple as using the _reindex API :

We can see here that we use the source filtering to index only the message field of the documents present in the dead letter index.

It could be useful to mark the document as fixed in the dead letter index or even delete it to indicate that there is no more error to handle on this one.