Importing 100M+ Records into DynamoDB in Under 30 Minutes | by Paul Singman | Towards Data Science

No longer will anyone suffer while setting up the process of doing a full export of a DynamoDB table to S3.
The same cannot be said, however, for someone looking to import data into a Dynamo table. Particularly a large amount of data and fast.
The need for quick bulk imports can occur when records in a table get corrupted, and the easiest way to fix them is do a full table drop-and-recreate. Or when streaming data into a table, it can be useful to run a nightly batch “true-up” job to correct any intra-day anomalies that may have occurred.
Reducing the time to completion for these types of jobs from over an hour to under a minute gives you more confidence in the state of your data at any given point in time. And it provides a more stable foundation for low-latent applications to be built on top of your data.
Say we have data in an S3 bucket at point A, and we want to move it into a Dynamo table at point B... how do we do this?
Well, the simplest approach is to write a little script that reads the file into a DataFrame object, loops through that object’s rows, and then does a dynamo.Table().batch_writer().put_item() operation to insert items into the table.
Running this script on a 500k row DataFrame results in the following write performance:
As you can see, the number of writes per second (w/s) to the table gets capped at about 650 w/s. Amazon indicates that the true limit of a single-process writing to Dynamo is 1,000 w/s, but due to network and other overhead, the observed value is lower (running locally from my laptop).
For 500,000 records, it takes about 15 minutes to insert all the records into the table. Depending on your use case, this might be acceptable. But for bulk inserts of larger datasets — say 100 million rows(!) — it’s clear that improving the write throughput rate would be helpful.
Can we do better?
To increase write performance it is clear there needs to be more than one process writing to the table. So how do we enable multiple processes to write in parallel?
The most elegant way I’ve found to achieve this is to invoke multiple Lambda Function writer instances simultaneously. Each one picks up a slice of the dataset and writes it to the Dynamo table in parallel.
The way this works is by setting a Lambda function to trigger off an SQS queue. This means that if a message is placed on the queue, the Lambda function will invoke and execute its code — taking the SQS message value as a function input parameter.
To go a step further — if such a Lambda were to contain code like in the simple_dynamo_write.py snippet above, and the message placed on the queue is an S3 filepath pointer to the data we want written, then it should be clear how this data will make it’s way into Dynamo!
Each Lambda instance will write to the table at 750-to-1000 w/s, so once we get multiple going at the same time, we can achieve write speeds much higher!
The trick is to split up and organize the data in such a way that Lambda can process it efficiently without error. There are AWS-imposed limits to how many Lambdas can be invoked simultaneously (1,000), how long a single Lambda invocation can run before it shuts down (15 min), and how many writes per second a single Dynamo table can handle (40,000).
It’s always smart to know the relevant quotas when working with a service, and in general AWS does a good job of documenting and making this information available.
With these limits in mind, it should be clear that with full parallelization we cannot distribute the dataset into more than 1000 SQS messages — otherwise we’ll hit the limit of 1000 simultaneous Lambda invocations.
Each file also can’t be too big, otherwise it’ll take more than 15 minutes for a Lambda to write it to the table (as a reminder, it took nearly 15 minutes to write a 500,000 row file earlier). It would not be good for a Lambda to shut down mid-file, since there would be no way to guarantee all rows get written without re-processing the entire file.
The way to control for all of these factors is to create a maximum of 40 SQS messages composed of comma-separated strings of the data’s S3 filepaths.
For example, if we have 3 files in S3 to write into Dynamo, we could create an SQS message like:
's3://bucket/file1.csv,s3://bucket/file2.csv,s3://bucket/file3.csv'
With this message as an input, what the Lambda does is:
Pop off the last filepath from the input string.Write that file’s data into the Dynamo table.Create a new message on the SQS queue that triggers itself with only the unprocessed files remaining.
Pop off the last filepath from the input string.
Write that file’s data into the Dynamo table.
Create a new message on the SQS queue that triggers itself with only the unprocessed files remaining.
So after writing one file, the new SQS message created will look like the below, and crucially a new instance of the Lambda function will almost immediately invoke to start processing file2.
's3://bucket/file1.csv,s3://bucket/file2.csv'
And once file2 gets processed, a new SQS message gets placed on the queue:
's3://bucket/file1.csv'
And after file1 gets written, no more queue messages are generated, no more Lambdas get invoked, and there’s no more data not in the Dynamo table!
To give a more tangible image of this recursive strategy, here’s an example Lambda function code:
Now, instead of the bounded write capacity metrics graph, you’ll be seeing more exciting visuals like this!
With this architecture, we can achieve writes per second speeds of up to 40k into Dynamo, since up to 40 processes can run in parallel, each writing at 1k rows per second.
Whereas before a 100M row dataset would take 40 hours at 1,000 w/s, at the increased rate we can import the full dataset in just 40 minutes! (As an aside, the 40k max write speed limit on a Dynamo table is simply the AWS default, and can be increased upon request).
In working with Dynamo as the backend for several data-intensive applications, I’ve found it extremely useful to have the ability to refresh the full dataset quickly in this way. In situations where specific rows may have become corrupted or mis-processed, more surgical approaches to fixing the data can be error-prone and time consuming, and are not recommended.
Lastly, there are a couple other factors I didn’t get a chance to cover in the article, like ideal handling of the Dynamo Capacity mode, and how related processes like Dynamo streams are impacted.
If curious about these, ask in the comments below or reach out on Twitter.
Thank you to Ryan Kelly and Michael Petryszyn for inspiration for this article.