Package: requests-ratelimiter
Requests-RatelimiterThis package is a simple wrapper aroundpyrate-limiter v2that adds convenient integration with therequestslibrary.Full project documentation can be found atrequests-ratelimiter.readthedocs.io.Featurespyrate-limiteris a general-purpose rate-limiting library that implements the leaky bucket
algorithm, supports multiple rate limits, and has optional persistence with SQLite and Redis
backendsrequests-ratelimiteradds some conveniences for sending rate-limited HTTP requests with therequestslibraryIt can be used as either asessionor atransport adapterIt can also be used as a mixin, for compatibility with otherrequests-based librariesRate limits are tracked separately per hostDifferent rate limits can optionally be applied to different hostsInstallationpip install requests-ratelimiterUsageUsage OptionsThere are three ways to userequests-ratelimiter:SessionThe simplest option isLimiterSession,
which can be used as a drop-in replacement forrequests.Session.Example:fromrequests_ratelimiterimportLimiterSessionfromtimeimporttime# Apply a rate limit of 5 requests per second to all requestssession=LimiterSession(per_second=5)start=time()# Send requests that stay within the defined rate limitforiinrange(20):response=session.get('https://httpbin.org/get')print(f'[t+{time()-start:.2f}] Sent request{i+1}')Example output:[t+0.22]Sentrequest1[t+0.26]Sentrequest2[t+0.30]Sentrequest3[t+0.34]Sentrequest4[t+0.39]Sentrequest5[t+1.24]Sentrequest6[t+1.28]Sentrequest7[t+1.32]Sentrequest8[t+1.37]Sentrequest9[t+1.41]Sentrequest10[t+2.04]Sentrequest11...AdapterFor more advanced usage,LimiterAdapteris available to be used as atransport adapter.Example:fromrequestsimportSessionfromrequests_ratelimiterimportLimiterAdaptersession=Session()# Apply a rate-limit (5 requests per second) to all requestsadapter=LimiterAdapter(per_second=5)session.mount('http://',adapter)session.mount('https://',adapter)# Send rate-limited requestsforuser_idinrange(100):response=session.get(f'https://api.some_site.com/v1/users/{user_id}')print(response.json())MixinFinally,LimiterMixinis available for advanced use cases in which you want add rate-limiting features to a custom session
or adapter class. SeeCustom Session Examplebelow for an example.Rate Limit SettingsBasic SettingsThe following parameters are available for the most common rate limit intervals:per_second: Max requests per secondper_minute: Max requests per minuteper_hour: Max requests per hourper_day: Max requests per dayper_month: Max requests per monthburst: Max number of consecutive requests allowed before applying per-second rate-limitingAdvanced SettingsIf you need to define more complex rate limits, you can create aLimiterobject instead:frompyrate_limiterimportDuration,RequestRate,Limiterfromrequests_ratelimiterimportLimiterSessionnanocentury_rate=RequestRate(10,Duration.SECOND*3.156)fortnight_rate=RequestRate(1000,Duration.DAY*14)trimonthly_rate=RequestRate(10000,Duration.MONTH*3)limiter=Limiter(nanocentury_rate,fortnight_rate,trimonthly_rate)session=LimiterSession(limiter=limiter)Seepyrate-limiter docsfor moreLimiterusage details.BackendsBy default, rate limits are tracked in memory and are not persistent. You can optionally use either
SQLite or Redis to persist rate limits across threads, processes, and/or application restarts.
You can specify which backend to use with thebucket_classargument. For example, to use SQLite:frompyrate_limiterimportSQLiteBucketfromrequests_ratelimiterimportLimiterSessionsession=LimiterSession(per_second=5,bucket_class=SQLiteBucket)Seepyrate-limiter docsfor more details.Other FeaturesPer-Host Rate Limit TrackingWith eitherLimiterSessionorLimiterAdapter, rate limits are tracked separately for each host.
In other words, requests sent to one host will not count against the rate limit for any other hosts:session=LimiterSession(per_second=5)# Make requests for two different hostsfor_inrange(10):response=session.get(f'https://httpbin.org/get')print(response.json())session.get(f'https://httpbingo.org/get')print(response.json())If you have a case where multiple hosts share the same rate limit, you can disable this behavior
with theper_hostoption:session=LimiterSession(per_second=5,per_host=False)Per-Host Rate Limit DefinitionsWithLimiterAdapter, you can apply different rate limits to different hosts or URLs:# Apply a different set of rate limits (2/second and 100/minute) to a specific hostadapter_2=LimiterAdapter(per_second=2,per_minute=100)session.mount('https://api.some_site.com',adapter_2)Behavior for matching requests is the same as other transport adapters:requestswill use the
adapter with the most specific (i.e., longest) URL prefix that matches a given request. For example:session.mount('https://api.some_site.com/v1',adapter_3)session.mount('https://api.some_site.com/v1/users',adapter_4)# This request will use adapter_3session.get('https://api.some_site.com/v1/')# This request will use adapter_4session.get('https://api.some_site.com/v1/users/1234')Rate Limit Error HandlingSometimes, server-side rate limiting may not behave exactly as documented (or may not be documented
at all). Or you might encounter other scenarios where your client-side limit gets out of sync with
the server-side limit. Typically, a server will send a429: Too Many Requestsresponse for an
exceeded rate limit.When this happens,requests-ratelimiterwill adjust its request log in an attempt to catch up to
the server-side limit. If a server sends a different status code other than 429 to indicate an
exceeded limit, you can set this withlimit_statuses:session=LimiterSession(per_second=5,limit_statuses=[429,500])Or if you would prefer to disable this behavior and handle it yourself:session=LimiterSession(per_second=5,limit_statuses=[])CompatibilityThere are many other useful libraries out there that add features torequests, most commonly by
extending or modifyingrequests.Sessionorrequests.HTTPAdapter.To userequests-ratelimiterwith one of these libraries, you have a few different options:If the library provides a customSessionclass, mount aLimiterAdapteron itOr useLimiterMixinto create a customSessionclass with features from both librariesIf the library provides a customAdapterclass, useLimiterMixinto create a customAdapterclass with features from both librariesCustom Session Example: Requests-CacheFor example, to combine withrequests-cache, which
also includes a separate mixin class:fromrequestsimportSessionfromrequests_cacheimportCacheMixin,RedisCachefromrequests_ratelimiterimportLimiterMixin,RedisBucketclassCachedLimiterSession(CacheMixin,LimiterMixin,Session):"""Session class with caching and rate-limiting behavior. Accepts arguments for bothLimiterSession and CachedSession."""# Optionally use Redis as both the bucket backend and the cache backendsession=CachedLimiterSession(per_second=5,bucket_class=RedisBucket,backend=RedisCache(),)This example has an extra benefit: cache hits won't count against your rate limit!