Getting Started with DynamoDB in .NET Core — How to Build a Leaderboard
By: Matthew Harper
Tags: Dynamodb, Programming, AWS, Technology, Dotnet

Getting Started with DynamoDB in .NET Core — How to Build a Leaderboard Matthew Harper Follow Jan 21 · 6 min read

If you’ve spent years working with relational databases, getting started with Amazon’s DynamoDB may feel intimidating to you. Traditional relational databases were optimized for storage; DynamoDB is optimized for performance. Tables exist independently; there is no ‘Join’ clause to query records from more than one table at a time. Concepts like normalization (not storing duplicate data) are no longer critical; in fact, sometimes your data model will explicitly duplicate data to improve performance.

The goal of this article is to learn the fundamentals of DynamoDB by diving right in and using it. We will design a DynamoDB schema to track the high scores for many users across multiple games. We’ll also use a .NET Core console app to populate and test our database.

Setup

If you’d prefer to dive straight into the DB schema and the source code, everything you need is on this GitHub repo. You’ll find a Terraform file that describes our finished DynamoDB table schema. If you haven’t used Terraform before, it’s a fantastic tool that allows you to write your infrastructure as code.

The only prerequisite required to use Terraform is that you have shared AWS credentials stored on your machine (if you’ve ever used the AWS CLI, you’re good.) Then download Terraform and add it to your path. Edit the example.tf file to change ‘profile’ to the name of your own AWS profile. Then open a command prompt in the directory containing the .tf file and run the following two commands:

Terraform will execute all the AWS APIs required to generate your infrastructure. If you make changes to your terraform file, you can run ‘terraform apply’ again to execute the updates. When you’re done using it, executing ‘terraform destroy’ will remove all your changes from your AWS environment.

The included .NET Core console app uses the AWS SDK to connect to our DynamoDB table, fill it with random data, and execute queries against it.

DynamoDB

To successfully implement DynamoDB, you need to first understand the desired access pattern. What problems are we trying to solve? How will our applications query our database?

What are the high scores for a user across all games?

What is the high score for a user for a specific game?

What is the most recent high score a user has achieved?

What is the overall high score for a game_id and what user achieved that score?

We’ll track four attributes in our table:

Username (string)

Game (string)

TopScore (number)

Timestamp (string)

Query vs Scan

Before starting on the schema, it’s important to understand the two ways we typically interact with our database. We can scan a table, which means examining every single row to determine if that record fits within the filter we are applying. Or we can query a table, which uses indexes (we’ll discuss below) to examine only a small range of items. When designing our database, the goal is usually to avoid scans on anything but the smallest tables because they can be very slow and can even use up all of your table’s read capacity for a short period (causing other requests to fail.)

Primary Key

Every DynamoDB table requires a Primary Key, and that key must be unique within the table. A Primary Key can be Partition Key, or a combination of a Partition Key and a Sort Key. We’ll define each of those terms below as we walk through our use cases.

Partition Key

DynamoDB is a distributed data store. This means the data in our table is not stored on a single location on disk, or even on a single disk. The way the data is organized is based on the partition key (also known as a hash key). Behind the scenes, DynamoDB assigns a number of partitions to your table. Whenever a new record is inserted, the partition key is sent through a hash function to determine which partition the new record will be stored in.

An ideal partition key will have a large range of inputs and a randomized access pattern. We don’t want most of our records going to the same partition, because our query performance will slow (too much data to scan) and most of our users’ queries will hit the same partition (again losing the benefits of the distributed nature of DynamoDB.) Aim for a partition key that allows even distribution of data and equal amounts of client queries across partitions.

In our example, Username would make a good partition key, because it provides a varied range of unique entries. But, since we want to store a high score for each Username and Game combination, Username will not be unique within our table and this will not satisfy the requirements of a primary key. We’re going to need a sort key.

Query by Partition Key

Sort Key

By specifying a Sort Key, the table’s Primary Key becomes a composite of the Partition Key and Sort Key. Within each partition, records are now stored in sorted order. This allows more complex queries to be executed against the table.

For example, we can ask for results that match a specific value for the sort key, items that are greater or less than a value, or items within a range of values.

Using Game as our sort key resolves our first and second use case. We now have a unique primary key (composite of Username and Game) and can query on just the partition key to find each of the top scores for a specific user. We can also query the high score for a specific Username and Game combination.

Query by Partition Key and Sort Key

Local Secondary Index

For our next use case, we want to know what the most recent high score is that a user achieved. But since our Sort Key is Game, we don’t have an efficient way to filter the results by timestamp. We could use a Scan instead of a Query, but that means looking at every entry in the table and then applying the filter. This is slow and, depending on the size of the table, can potentially use up all of your provisioned read capacity.

We basically need another sort key — luckily, DynamoDB provides this in the form of a Local Secondary Index. LSIs, for short, can only be defined at table creation; they cannot be added later. An LSI uses the same partition key that is already defined but allows you to choose a new sort key. In our scenario, we’ll use Timestamp as our LSI’s sort key. Now we can query against our new index to return the most recent high score. Note that we use ScanIndexForward=false to sort the results in descending order and Limit=1 to return a single result.

Query against Local Secondary Index

Global Secondary Index

Now we have a challenge — find the high score recorded by any user for a specific game. Because Username is our Partition Key, high scores for different users are stored in different locations, making it difficult for us to effectively search for a global high score. We’d need to retrieve the top scores from each partition and sort them all into one list — an expensive operation.

A better way to approach this is to use a Global Secondary Index (or GSI.) You can create one at any time, even after table creation, by specifying a new Partition Key and optionally a Sort Key. Both of these keys can differ from the ones used to originally create the table. We’re effectively creating a duplicate version of our table but organizing the data differently. Along with the keys, we can choose what other attributes of the table are included in this new index. To reduce storage and read/write capacity, these attributes should be limited to only what is necessary for your use case.

To find the global high score for a game, we need all of the scores to be in the same partition. So we’ll create a GSI with the Game as the Partition Key and TopScore as the Sort Key.

Query against Global Secondary Index

Recap

DynamodDB is not a fit for every use case, but it excels in situations where you need key-value lookups or can take advantage of its partitions to do queries on a small range of records. The fact that it’s a managed, serverless AWS offering also makes it appealing to developers. It also integrates nicely with AWS Lambda; in fact, these two services often make up the back end of an Alexa Skill.

DynamoDB also supports Streams, a feature that we did not discuss here. Streams allow changes to your table to be captured for consumption by another service. For example, you could send an email to a user whenever they achieve a new high score. To explore how to use set up a Stream and connect it to Lambda, check out my next article.

The code and the terraform for these examples is available on GitHub. Please share feedback and questions in the comments. Thanks!