Package: self-limiters
Self limitersA library for regulating traffic with respect toconcurrencyortime.It implements asemaphoreto be used when you need to
limit the number of concurrent requests to an API (or other resources). For example if you can only
send 5 concurrent requests.It also implements thetoken bucket algorithmwhich can be used
to limit the number of requests made in a given time interval. For example if you're restricted to 10 requests
per second.Both limiters are async, FIFO, and distributed using Redis. You should probably only use this if you need
distributed queues.This was written with rate-limiting in mind, but the semaphore and token bucket
implementations can be used for anything.Installationpip install self-limitersUsageBoth implementations are written as async context managers.SemaphoreTheSemaphorecan be used like this:fromself_limitersimportSemaphore# 5 requests at the timeasyncwithSemaphore(name="",capacity=5,max_sleep=60,redis_url=""):client.get(...)We useblpopto wait for the semaphore to be freed up, under the hood, which is non-blocking.If you specify a non-zeromax_sleep, aMaxSleepExceededErrorwill be raised ifblpopwaits for longer than that specified value.Token bucketTheTokenBucketcontext manager is used the same way, like this:fromself_limitersimportTokenBucket# 1 requests per minuteasyncwithTokenBucket(name="",capacity=1,refill_amount=1,refill_frequency=60,max_sleep=600,redis_url=""):client.get(...)The limiter first estimates when there will be capacity in the bucket - i.e., when it's this instances turn to go,
then async sleeps until then.Ifmax_sleepis set and the estimated sleep time exceeds this, aMaxSleepExceededErroris raised immediately.As a decoratorThe package doesn't ship any decorators, but if you would
like to limit the rate at which a whole function is run,
you can create your own, like this:fromself_limitersimportSemaphore# Define a decorator functiondeflimit(name,capacity):defmiddle(f):asyncdefinner(*args,**kwargs):asyncwithSemaphore(name=name,capacity=capacity,redis_url="redis://127.0.0.1:6389"):returnawaitf(*args,**kwargs)returninnerreturnmiddle# Then pass the relevant limiter arguments like this@limit(name="foo",capacity=5)deffetch_foo(id:UUID)->Foo:Implementation and performance breakdownThe library is written in Rust (for fun) and relies onLuascripts andpipeliningto
improve the performance of each implementation.Redis lets users upload and execute Lua scripts on the server directly, meaning we can write
e.g., the entire token bucket logic in Lua. This present a couple of nice benefits:Since they are executed on the redis instance, we can make 1 request to redis
where we would otherwise have to make 3 or 4. The time saved by reducing
the number of requests is significant.Redis is single-threaded and guarantees atomic execution of scripts, meaning
we don't have to worry about data races. In a prior iteration, when we had to make 4 requests
to estimate the wake-up time for a token bucket instance, we had needed to use theredlockalgorithm to ensure fairness. With Lua scripts, our implementations are FIFO out of the box.So in summary they make our implementation faster, since we save several round-trips to the server
and backandsince we no longer need locks, and distributed locks are expensive. And they
simultaneously make the code much, much simpler.This is how each implementation has ended up looking:The semaphore implementationRun alua scriptto create a list data structure in redis, as the foundation of the semaphore.This script is idempotent, and skipped if it has already been created.RunBLPOPto non-blockingly wait until the semaphore has capacity, and pop from the list when it does.Then run apipelined commandto release the semaphore by adding back the capacity.So in total we make 3 calls to redis, where we would have made 6 without the scripts, which are all non-blocking.The token bucket implementationThe token bucket implementation is even simpler. The steps are:Run alua scriptto estimate and return a wake-up time.Sleep until the given timestamp.We make 1 call instead of 3, then sleep. Both are non-blocking.In other words, the very large majority of time is spent waiting in a non-blocking way, meaning the limiters' impact on an application event-loop should be close to completely negligible.BenchmarksWe run benchmarks in CI with Github actions. For a normalubuntu-latestrunner, we see runtimes for both limiters:When creating 100 instances of each implementation and calling them at the same time, the average runtimes are:Semaphore implementation: ~0.6ms per instanceToken bucket implementation: ~0.03ms per instanceTake a look at thebenchmarking scriptif you want
to run your own tests!Implementation referenceThe semaphore implementationThe semaphore implementation is useful when you need to limit a process
tonconcurrent actions. For example if you have several web servers, and
you're interacting with an API that will only tolerate a certain amount of
concurrent requests before locking you out.The flow can be broken down as follows:The initiallua scriptfirst checks if the redis list we will build the semaphore on exists or not.
It does this by callingSETNXon the key of the queue plus a postfix
(if thenamespecified in the class instantiation is "my-queue", then the queue name will be__self-limiters:my-queueand setnx will be called for__self-limiters:my-queue-exists). If the returned
value is 1 it means the queue we will use for our semaphore does not exist yet and needs to be created.It might strike you as weird to maintain a separate value, just to indicate whether a list exists,
when we could just check the list itself. It would be nice if we could useEXISTSon the list directly, but unfortunately a list is considered
not to exist when all elements are popped (i.e., when a semaphore is fully acquired), so I don't see
another way of doing this. Contributions are very welcome if you do!Then if the queue needs to be created we callRPUSHwith the number of arguments
equal to thecapacityvalue used when initializing the semaphore instance. For a semaphore with
a capacity of 5, we callRPUSH 1 1 1 1 1, where the values are completely arbitrary.Once the list/queue has been created, weBLPOPto block until it's
our turn.BLPOPis FIFO by default. We also make sure to specify themax_sleepbased on the initialized
semaphore instance setting. If nothing was passed we allow sleeping forever.On__aexit__we run three commands in a pipelined query. WeRPUSHa1back into the queue to "release" the semaphore, and set an expiry on the queue and the string value we calledSETNXon.The expires are a half measure for dealing with dropped capacity. If a node holding the semaphore dies,
the capacity might never be returned. If, however, there is no one using the semaphore for the duration of the
expiry value, all values will be cleared, and the semaphore will be recreated at full capacity next time it's used.
The expiry is 30 seconds at the time of writing, but could be made configurable.The token bucket implementationThe token bucket implementation is useful when you need to limit a process by
a time interval. For example, to 1 request per minute, or 50 requests every 10 seconds.The implementation is forward-looking. It works out the time therewould have beencapacity in the bucket for a given client and returns that time. From there we can
asynchronously sleep until it's time to perform our rate limited action.The flow can be broken down as follows:Call theschedule Lua scriptwhich firstGETs thestateof the bucket.The bucket state contains the last time slot scheduled and the number of tokens left for that time slot.
With a capacity of 1, having atokens_left_for_slotvariable makes no sense, but if there's
capacity of 2 or more, it is possible that we will need to schedule multiple clients to the
same time slot.The script then works out whether to decrement thetokens_left_for_slotvalue, or to
increment the time slot value wrt. the frequency variable.Finally, we store the bucket state again usingSETEX.
This allows us to store the state and set expiry at the same time. The default expiry
is 30 at the time of writing, but could be made configurable.One thing to note, is that this would not work if it wasn't for the fact that redis is single threaded,
so Lua scripts on Redis are FIFO. Without this we would need locks and a lot more logic.Then we just sleep!ContributingPlease do! Feedback on the implementation, issues, and PRs are all welcome. SeeCONTRIBUTING.mdfor more details.Please also consider starring the repo to raise visibility.