Rate Limiting a .NET Core Application using the Sliding Window Algorithm — Part 1, Implementing the Sliding Window Algorithm
Tags: Networking, Csharp, Algorithms, Dotnet Core, Rate Limiting

Rate Limiting a .NET Core Application using the Sliding Window Algorithm — Part 1, Implementing the Sliding Window Algorithm Gopherchucks Follow Oct 12, 2020 · 12 min read

Rate limiting prevents an API from being saturated with requests, ensuring that the service provided is not degraded. Rate limiting is a tactic commonly employed by an API but can also be employed by a client that consumes from an API. We might see the latter scenario for an application consuming from a third-party API that imposes request-rate restrictions with punitive measures for consumers routinely in violation of the given restrictions. Recently, I encountered such a scenario.

Figure 1: A visualization of a rate-limited API client

Precedence for a Rate-Limited API Client

Some time ago, I started work on a personal project: a web app that displays match statistics about highly ranked players in a competitive video game. The statistics displayed on the web app are produced by a .NET Core console app. The console app runs on an hourly interval, pulling the latest match data from the game’s public API, aggregating the data, and subsequently storing it for later use by the web app. The game’s API imposes request-rate restrictions with the typical limitations seen for a public API: a consumer should not exceed X requests over Y seconds. Applications that are routinely in violation of the rate limit are blocked from accessing the API. Given that my console app attempts to pull all of the latest match data every hour, as quickly as possible, I set out to implement a rate limiter for the app to prevent it from being penalized by the API.

The Structure of this Series of Articles

In the following series of articles, I will document how I implemented a rate limiter for a .NET Core application. Each article in this series will have a separate focus,

The first article (what you are currently reading) puts focus on the rate limiter algorithm being used and how it can be implemented in .NET Core

The second article documents how the implementation of the rate limiter can be validated with unit tests

The third article provides additional analysis of the chosen rate limiter algorithm, and data from several simulations that make use of the rate limiter implementation

The last article details how the rate limiter implementation can be used by a .NET Core application to limit outbound requests to an API

The implementation I am about to discuss was developed with the intent of being used within a single application — and does not apply in a distributed setting. This is not to say that the algorithm could not be applied in a distributed setting or that the below code could not be modified to handle such a use case, but that it is out of the scope of this series of articles.

Choosing a Rate Limiting Algorithm

Our first order of business is choosing what algorithm to implement our rate limiter with. A rudimentary approach would involve tracking the number of requests over an interval of time (corresponding to the interval of the rate limit) and resetting the request counter when a new interval has started. Unfortunately, this approach (commonly named the fixed window algorithm) fails to protect against bursts of traffic that occur at the end of the previous sampling period* and the beginning of the current. This approach is not appropriate for our purposes as we expect to output a high rate of traffic and need to avoid allowing non-conforming requests to get past our rate limiter.

Fortunately for us, there are myriad of proven rate limiter algorithms to choose from, however, this article is going to be of considerable length so I will not get into the details of each of the prospective algorithms (some excellent articles already do this). Instead, I swill spare you the details and tell you the algorithm I opted for: the sliding window.

I felt the sliding window algorithm was the optimal choice for several reasons,

The concept is simple to intuit

It executes quickly

It has a low memory footprint

It protects against bursts at the boundaries of two adjacent sampling periods (the problem unaccounted for with the rudimentary approach)

It does not involve the management of queues or timers, keeping complexity to a minimum

*A sampling period is the unit of time for a rate limit rule (e.g. 1 second in a 100 request/second rule).

The Sliding Window Algorithm

Now that we have established what algorithm we will be using, we can dive into the details on how it works. To better communicate the algorithm, we will henceforth refer to sampling periods as windows. The algorithm operates as follows,

The request counts for the current and previous windows are tracked, being reset or replaced as the boundaries for these windows are exceeded and new windows take their respective places

When an incoming request is received, the request is checked to see if it conforms; a request is checked for conformity by calculating the approximate request count of the two windows being tracked, and comparing the approximate count to the prescribed rate limit

The approximate request count is a combination of the current request count and a weighted value of the previous request count that is proportional to the time elapsed in the current window

The approximate request count be thought of as being part of an approximate window: our best effort at determining the true request rate. As time elapses, the windows being tracked are ever-shifting and consequentially, so too is the approximate window — hence the term, “sliding window”.

Visualizing the Sliding Window Algorithm

As the above details are somewhat abstract, a visual is provided below to help clarify. The windows in the visual are spaced apart from one another to distinguish them as separate entities; but, in reality, their boundaries are touching as there can be no gaps in time.

Figure 2: A visualization of the approximate window that is calculated in the sliding window algorithm

In the visualization above, we see the approximate window displayed as a dashed square, overlaying the actual windows being tracked. Using this visual, we see that the approximate window clearly outlines what proportion each of the two windows being tracked contributes to the approximate request count.

Calculating the Approximate Request Count

Now that we know how the algorithm functions, we need an equation for calculating the approximate request count,

Figure 3: An equation for calculating the approximate request count

RCa is the approximate request count

is the approximate request count RCp is the request count of the previous window

is the request count of the previous window Tw is the prescribed length of time for a window

is the prescribed length of time for a window Te is the time elapsed in the current window

is the time elapsed in the current window RCc is the request count of the current window

From the equation, we see that the previous request count’s contribution is proportional to the time elapsed in the current window.

Implementing the Sliding Window Algorithm by Scenario

Now that we know how the algorithm operates, we can discuss the implementation. I have broken-down the implementation details into several sub-sections. The first two sub-sections include some initial setup while the remaining sub-sections are dedicated to the various scenarios that can occur for the algorithm. I find that breaking the algorithm down by scenario makes it easier to reason about the code.

Getting the Time

We need to be able to retrieve precise measurements of time so that we may properly segment the windows we track and confidently categorize incoming requests into said windows. As this is a .NET Core implementation, the static method call, Stopwatch.GetTimestamp() of System.Diagnostics fits the bill:

If supported, the method call will use a high-resolution performance counter that relies on the underlying system

If the high-resolution performance counter is not supported, the method call will default to using System.DateTime.Ticks

Now that we know how to get a precise measurement of time, we can move on to the next section, right? Wrong! As every good programmer does, we will be writing unit tests to validate our code. Unfortunately, it can be difficult to write unit tests for code that has functionality dependent on time. Thankfully for us, there is a solution:

We can write an interface, ITimestamp , that contains a method for returning the time and an implementation that provides the time by utilizing the Stopwatch.GetTimestamp() operation

, that contains a method for returning the time and an implementation that provides the time by utilizing the operation In the class that will represent our implementation of the sliding window algorithm, we will inject the ITimestamp implementation as a dependency of the constructor and use the class as the sole source of measurements of time

implementation as a dependency of the constructor and use the class as the sole source of measurements of time When it comes time (pun intended) to write unit tests, we will mock our ITimestamp implementation and precisely control what values are returned to the sliding window implementation

The ITimestamp interface and its corresponding implementation are documented below,

We use a return type of long for the interface as ticks are the currency of the Stopwatch class and we want to ensure that we maintain the resolution of the data returned.

Setting up the SlidingWindow Class

We can now turn our attention towards the implementation of the sliding window algorithm, the SlidingWindow class. The class is relatively simple, consisting of,

A singular constructor with input parameters for the request count limit, rate limit interval, and the ITimestamp interface that we designed above

interface that we designed above Several private properties for tracking the request count of the current and previous windows, and the time elapsed since the start of the current window

A method, RequestConforms() that returns a boolean indicating whether or not the incoming request conforms to the provided rate limit

From the gist above, we see the various private properties that were previously mentioned,

_syncObject is an arbitrary object for synchronizing concurrent requests to the RequestConforms() method

is an arbitrary object for synchronizing concurrent requests to the method _timestamp will contain our implementation of the ITimestamp interface and will be the sole provider of measurements of time

will contain our implementation of the interface and will be the sole provider of measurements of time _requestIntervalTicks represents the interval of time (in ticks) for the rate limit and consequentially, the length of time occupied by the windows we will be tracking

represents the interval of time (in ticks) for the rate limit and consequentially, the length of time occupied by the windows we will be tracking _requestLimit represents the number of requests allowed for a given rate limit interval

represents the number of requests allowed for a given rate limit interval _windowStartTime is a nullable value representing the start time (in ticks) of the current window — or in the case of a null, that a request has yet to be received by the rate limiter

is a nullable value representing the start time (in ticks) of the current window — or in the case of a null, that a request has yet to be received by the rate limiter _prevRequestCount represents the number of conforming requests in the previous window

represents the number of requests in the previous window _requestCount represents the number of conforming requests in the current window

Bringing our attention to the constructor, there are two things to note,

We added some validation on the values provided for requestLimit and requestIntervalMs to prevent the user from supplying erroneous values for the rate limit

and to prevent the user from supplying erroneous values for the rate limit From the naming convention, we clearly indicate that the value for requestIntervalMs should be in milliseconds, but then immediately convert it to ticks in the constructor. This is done because we need the value to be in ticks for our calculations, but it is much easier to reason about a rate limit in terms of milliseconds

With this, we now have everything we need to begin implementing the algorithm. Again, the implementation is broken down by the possible scenarios that could occur for the algorithm as incoming requests are received.

Scenario #1: First Request

Figure 4: A visualization of the sliding window algorithm when the first request is received

Starting with our first scenario, we have the simplest situation to handle: the incoming request is the first request seen by the rate limiter. In this scenario, we have not tracked a previous window, so our approximate request count becomes the request count of the current window.

The resulting code for RequestConforms() will be as follows,

As you are seeing the contents of the RequestConforms() method for the first time, there are several things to note,

We immediately lock on _syncObject before performing any logic. We do this because we expect concurrent calls and need to prevent synchronization issues as we modify the values being tracked

before performing any logic. We do this because we expect concurrent calls and need to prevent synchronization issues as we modify the values being tracked We get the current time within the lock. If we were to do this outside of the lock, we risk having an incorrect timestamp for when are actually able to evaluate the request (i.e. when we obtained the lock)

The value for elapsedTime is 0 if _windowStartTime is null . This is because we do not have a currently tracked window and therefore cannot have an elapsed time within the window

is if is . This is because we do not have a currently tracked window and therefore cannot have an elapsed time within the window We check that _windowStartTime has a value to indicate whether or not this is the first request

has a value to indicate whether or not this is the first request In the case that this is the first request, we set the value for _windowStartTime to the current time to indicate that we have started tracking the current window

to the current time to indicate that we have started tracking the current window Lastly, even if this is the first request, you may have noticed that we still calculate a value for the weightedRequestCount . This is not a mistake, the values for _prevRequestCount and elapsedTime will be 0 , so the resulting calculation will evaluate to the current window’s request count

Scenario #2: In Current Window with no Previous Window Tracked

Figure 5: A visualization of the sliding window algorithm when a request is received, but there is no previous window being tracked

For our second scenario, the incoming request is contained within the current window, and we have not tracked a previous window. This scenario will evaluate the same as the first: our approximate request count simply becomes the request count of the current window. No immediate code changes need to be made.

Scenario #3: In Current Window with a Previous Window Tracked

Figure 6: A visualization of the sliding window algorithm when a request is received and there is a previous window being tracked

With our third scenario, like the second scenario, the incoming request is contained within the current window; however, we now have a previous window tracked. When this occurs, the previous request count will now impact the value calculated for the weightedRequestCount . No code changes are made as we already utilize the _prevRequestCount and elapsedTime to calculate the weightedRequestCount .

Scenario #4: Exceeded the Current Window Since the Last Request

Figure 7: A visualization of the sliding window algorithm when the incoming request has exceeded the bounds of the current window

Our fourth scenario shakes things up a bit. In this scenario, our last request was tracked within the current window, but there was enough of a delay between the last request and the incoming request — that the incoming request exceeds the current window and now slots within the adjacent window.

To handle this, some code changes need to be made,

We set what was the current window as the previous window, updating the value for _prevRequestCount accordingly

accordingly We retroactively calculate when the new current window started and set that as the value for _windowStartTime

The value for _requestCount is reset and we need to recalculate the value for elapsedTime , using the new value for _windowStartTime

From the gist above, we see that,

We check that elapsedTime has exceeded _requestIntervalTicks , indicating that the current window has been exceeded

has exceeded , indicating that the current window has been exceeded We have an additional check to see if the current window has been exceeded by more than twice the window length; this will be explained in the fifth scenario

When the incoming request has exceeded the current window, but by less than two window lengths, we make all of the changes listed above

No other changes are made and we move on to calculate the value for the weightedRequestCount , using the adjusted values for elapsedTime , _prevRequestCount , and _requestCount

Scenario #5: Exceeded the Current Window by more than Two Windows Since the Last Request

Figure 8: A visualization of the sliding window algorithm when the incoming request has exceeded the current window by two window lengths

Our fifth and final scenario is an extension of the fourth scenario: the incoming request has exceeded the current window, but in this case, by more than two window lengths. We could perform the same operations that were done in the fourth scenario, but a simpler solution exists.

If you examine the visual above, you will see that the previous current window is not adjacent to what will become our new current window; an untracked window sits between the two. The untracked window is what would become the previous. As the untracked window has no requests tracked against it, our calculation for weightedRequestCount simply becomes the value of the current request count. We end up with the same result as the first scenario!

Knowing this, our final code change for RequestConforms() becomes,

From the gist above, you will see that we do not perform any calculations like we do in the fourth scenario. Instead, we make an optimization and simply reset the state.

Putting it all Together