How to Insert Records and Do Conditional Delete for DynamoDB Using Python and Boto3 Library | by billydharmawan | Towards Data Science

It is very common to have many junk or dummy records in our DynamoDB table for testing purposes. It may be an application we are developing or even just a function. It can also be some records that our existing application generates but, actually, they have no values. Regardless of what it is, we end up creating many records in our database.
Every now and then, we want to clean up our database tables, so only valuable and meaningful records are stored. For example, we did a load testing to test the user account creation API in our application. This obviously resulted in many records in the table, which are just “test” records. All these “test” records have the characteristics of the email starting with testing and last name starting with TEST. You know, whatever. 😆 These records, obviously, can be removed from the table once we finish our load testing.
In this tutorial, you will learn how to use Python to insert items and conditionally delete items from your DynamoDB table, which you do not want to keep anymore, for whatever reasons it may be. We are going to use the official AWS SDK library for Python, which is known as Boto3.
I am not going to do a detailed walkthrough on this but will show you quickly how to do it via virtualenv.
First things first, you need to install virtualenv via pip3 if you haven’t already. It is as simple as:
$ ~/demo > pip3 install virtualenv
Then, create a directory where you want to put your project file (or in this case, the Python script).
$ ~/demo > mkdir batch-ops-dynamodb
Go to the directory and create the virtual environment in there.
$ ~/demo > cd ./batch-ops-dynamodb$ ~/demo/batch-ops-dynamodb > virtualenv ./venvUsing base prefix '/Library/Frameworks/Python.framework/Versions/3.8'New python executable in /Users/billyde/demo/batch-ops-dynamodb/venv/bin/python3.8Also creating executable in /Users/billyde/demo/batch-ops-dynamodb/venv/bin/pythonInstalling setuptools, pip, wheel...done.
Lastly, we want to active the virtual environment. Just a quick reminder, a virtual environment is really useful as it isolates your development from the rest of your machine. Think of it like a container, where all the dependencies are provided within the virtual environment and accessible only from within the environment.
$ ~/demo/batch-ops-dynamodb > source ./venv/bin/activate$ ~/demo/batch-ops-dynamodb (venv) >
Notice the (venv). This indicates that you are in the virtual environment. You may not see that, though, depending on your terminal’s settings. So, as an alternative, here’s why you can validate whether your virtual environment is active or not.
$ ~/demo/batch-ops-dynamodb (venv) > which python/Users/billyde/demo/batch-ops-dynamodb/venv/bin/python$ ~/demo/batch-ops-dynamodb (venv) > which pip/Users/billyde/demo/batch-ops-dynamodb/venv/bin/pip
You can see that the Python and pip executables are the one from our virtual environment. All is good. 🙂
The only dependency for this tutorial is the Boto3 library. So, let’s go ahead and install it within our virtual environment.
$ ~/demo/batch-ops-dynamodb (venv) > pip install boto3Collecting boto3  Downloading boto3-1.11.7-py2.py3-none-any.whl (128 kB)     |████████████████████████████████| 128 kB 1.0 MB/sCollecting botocore<1.15.0,>=1.14.7  Downloading botocore-1.14.7-py2.py3-none-any.whl (5.9 MB)     |████████████████████████████████| 5.9 MB 462 kB/sCollecting s3transfer<0.4.0,>=0.3.0  Downloading s3transfer-0.3.1-py2.py3-none-any.whl (69 kB)     |████████████████████████████████| 69 kB 2.1 MB/sCollecting jmespath<1.0.0,>=0.7.1  Using cached jmespath-0.9.4-py2.py3-none-any.whl (24 kB)Collecting urllib3<1.26,>=1.20  Downloading urllib3-1.25.8-py2.py3-none-any.whl (125 kB)     |████████████████████████████████| 125 kB 5.3 MB/sCollecting python-dateutil<3.0.0,>=2.1  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)Collecting docutils<0.16,>=0.10  Using cached docutils-0.15.2-py3-none-any.whl (547 kB)Collecting six>=1.5  Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)Installing collected packages: urllib3, six, python-dateutil, docutils, jmespath, botocore, s3transfer, boto3Successfully installed boto3-1.11.7 botocore-1.14.7 docutils-0.15.2 jmespath-0.9.4 python-dateutil-2.8.1 s3transfer-0.3.1 six-1.14.0 urllib3-1.25.8
Once you have completed the pre-requisites, we’re ready to do the fun stuff, which is writing the actual script that will do the batch delete of our junk records.
To help us see how the script works, we will spin up a local DynamoDB instance in a Docker container. You can follow this tutorial to achieve this.
Essentially, what you need to do is spin up the DynamoDB Docker and create the table demo-customer-info as written in the tutorial.
Let’s create dummy records so we can see how the batch operation works. To do this, we are going to write a Python script that calls the DynamoDB PutItem operation in a loop. Create a new Python file in batch-ops-dynamo and name it insert_dummy_records.py.
~/demo/batch-ops-dynamodb ❯ touch insert_dummy_records.py
As mentioned in the Introduction section earlier, our dummy records will have the following traits:
last name begins with TEST
email address begins with testing
Our script will have the following components:
insert_dummy_record: a function that does the PutItem operation
for loop: a loop that will call insert_dummy_record function 10 times to insert dummy records.
We also leverage random.randint method to generate some random integers to be added to our dummy records’ attributes, which are:
customerId
lastName
emailAddress
Our script looks good! Now, it’s time for us to run it from the command line, using the Python executable from our virtual environment.
~/demo/batch-ops-dynamodb ❯ python3 insert_dummy_records.pyInserting record number 1 with customerId 769Inserting record number 2 with customerId 885Inserting record number 3 with customerId 873Inserting record number 4 with customerId 827Inserting record number 5 with customerId 231Inserting record number 6 with customerId 199Inserting record number 7 with customerId 272Inserting record number 8 with customerId 268Inserting record number 9 with customerId 729Inserting record number 10 with customerId 289
Ok. The script seems to work as expected. Hooray! 😄
Let’s validate by calling the scan operation on our local DynamoDB demo-customer-info table to check the records.
~/demo/batch-ops-dynamodb ❯ aws dynamodb scan --endpoint-url http://localhost:8042 --table-name demo-customer-info{    "Items": [        {            "customerId": {                "S": "199"            },            "lastName": {                "S": "TEST199"            },            "emailAddress": {                "S": "testing199@dummy.com"            }        },        {            "customerId": {                "S": "769"            },            "lastName": {                "S": "TEST769"            },            "emailAddress": {                "S": "testing769@dummy.com"            }        },... truncated... truncated... truncated        {            "customerId": {                "S": "827"            },            "lastName": {                "S": "TEST827"            },            "emailAddress": {                "S": "testing827@dummy.com"            }        }    ],    "Count": 10,    "ScannedCount": 10,    "ConsumedCapacity": null}
Perfect! We have some dummy records in our table now.
We’ll quickly insert 2 records that are “real” to the table. To do this, we are just going to write another Python script that takes command line argument as its input. Let’s name the file insert_real_record.py.
~/demo/batch-ops-dynamodb ❯ touch insert_real_record.py
The content of the file will be as follows.
Let’s go ahead and insert 2 records to the table.
~/demo/batch-ops-dynamodb ❯ python insert_real_record.py 11111 jones sam.jones@something.comInserting record with customerId 11111~/demo/batch-ops-dynamodb ❯ python insert_real_record.py 22222 smith jack.smith@somedomain.comInserting record with customerId 22222
Finally, we are going to write the script that deletes records that meet some specified conditions from our table.
Again, as a reminder, we want to remove the dummy records we inserted. The filter that we need to apply is last name begins with TEST and email address begins with testing.
How the script works:
do a scan operation on the table with the given filter expression
retrieve just the customerId attribute from all records, which meet our filter expression, as this is all we need to do the DeleteItem operation. Remember, customerId is the partition key of the table.
in a for loop, for each customerId returned by our scan operation, do the DeleteItem operation.
Go ahead and run this script from the command line.
~/demo/batch-ops-dynamodb ❯ python delete_records_conditionally.pyGetting customer ids to delete============['199', '769', '873', '268', '289', '231', '272', '885', '729', '827']Deleting customer 199Deleting customer 769Deleting customer 873Deleting customer 268Deleting customer 289Deleting customer 231Deleting customer 272Deleting customer 885Deleting customer 729Deleting customer 827
Great! The script works as intended. Now, let’s check the remaining records in our table.
~/demo/batch-ops-dynamodb ❯ aws dynamodb scan --endpoint-url http://localhost:8042 --table-name demo-customer-info{    "Items": [        {            "customerId": {                "S": "22222"            },            "lastName": {                "S": "smith"            },            "emailAddress": {                "S": "jack.smith@somedomain.com"            }        },        {            "customerId": {                "S": "11111"            },            "lastName": {                "S": "jones"            },            "emailAddress": {                "S": "sam.jones@something.com"            }        }    ],    "Count": 2,    "ScannedCount": 2,    "ConsumedCapacity": null}
There are only 2 records remaining, which are the “real” records we inserted. We can be certain now that our delete_records_conditionally.py script does what it is intended for.
Awesome stuff. 👍
Let’s take a look at a few things from the delete_records_conditionally.py script we just wrote.
Notice that we have a deserializer and we use it to get the customerId. The reason is because, the scan operation actually returns something like this.
# scan operation response{'Items': [{'customerId': {'S': '300'}}, {'customerId': {'S': '794'}}, {'customerId': {'S': '266'}}, {'customerId': {'S': '281'}}, {'customerId': {'S': '223'}}, {'customerId': {'S': '660'}}, {'customerId': {'S': '384'}}, {'customerId': {'S': '673'}}, {'customerId': {'S': '378'}}, {'customerId': {'S': '426'}}], 'Count': 10, 'ScannedCount': 12, 'ResponseMetadata': {'RequestId': 'eb18e221-d825-4f28-b142-ff616d0ca323', 'HTTPStatusCode': 200, 'HTTPHeaders': {'content-type': 'application/x-amz-json-1.0', 'x-amz-crc32': '2155737492', 'x-amzn-requestid': 'eb18e221-d825-4f28-b142-ff616d0ca323', 'content-length': '310', 'server': 'Jetty(8.1.12.v20130726)'}, 'RetryAttempts': 0}}
Since we are only interested in the value of customerId, we need to deserialise the DynamoDB item using the TypeDeserializer that is provided by the Boto3 library.
Another component worth mentioning is the Select and ProjectionExpression, which are parameters of the scan function. These 2 parameters work hand-in-hand. We set the value of Select to SPECIFIC_ATTRIBUTES and according to the official Boto3 documentation, this will return only the attributes listed in AttributesToGet. AttributesToGet has been marked as a legacy parameter and AWS advises us to use ProjectionExpression, instead.
From the official Boto3 documentation:
“If you use the ProjectionExpression parameter, then the value for Select can only be SPECIFIC_ATTRIBUTES . Any other value for Select will return an error.”
These 2 parameters are basically saying that the scan operation should only return the customerId attribute, which is what we need.
Lastly, we use the begins_with() function that is provided by AWS. This function takes an attribute name and the prefix to check against the value of the attribute specified.
By this point, you will have learnt how to do insert and delete DynamoDB records with Python and Boto3. You can certainly adjust and modify the script to suit your needs. For instance, the filter expression used in this tutorial is relatively simple, you can add more complex conditions, if need be.
Anyways, I hope this tutorial has given you a basic understanding of how to do DynamoDB operation using Python. So, go ahead and be creative, taking this script and enhance it to do more advanced stuff. Happy hacking! 🙂
Note: here’s the Github repo.