Package: requests-respectful
If you know Python, you know`Requests <http://docs.python-requests.org/>`__.Requestsis love.Requestsis life. Depending on your use cases, you may come across
scenarios where you need to useRequestsa lot. Services you consume
may have rate-limiting policies in place or you may just happen to be in
a good mood and feel like being a good Netizen. This is whererequests-respectfulcan come in handy.*requests-respectful*:Is a minimalist wrapper on top ofRequeststo work within rate
limits of any amount of services simultaneouslyCan scale out of a single thread, single process or even a single
machineEnables maximizing your allowed requests without ever going over set
limits and having to handle the falloutProxiesRequestsHTTP verb methods (for minimal code changes)Works with both Python 2 and 3 and is fully testedIs cool (hopefully?)Typical *requests* callimportrequestsresponse=requests.get("http://github.com",params={"foo":"bar"})Magic *requests-respectful* call-requestsverb methods are
proxied!fromrequests_respectfulimportRespectfulRequesterrr=RespectfulRequester()# This can be done elsewhere but the realm needs to be registered!rr.register_realm("Github",max_requests=100,timespan=60)response=rr.get("http://github.com",params={"foo":"bar"},realms=["Github"],wait=True)Conservative *requests-respectful* call- pass a lambda with arequestsmethod callimportrequestsfromrequests_respectfulimportRespectfulRequesterrr=RespectfulRequester()# This can be done elsewhere but the realm needs to be registered!rr.register_realm("Github",max_requests=100,timespan=60)request_func=lambda:requests.get("http://github.com",params={"foo":"bar"})response=rr.request(request_func,realms=["Github"],wait=True)RequirementsRedis> 2.8.0 (See FAQ if you are rolling your
eyes)Installationpipinstallrequests-respectfulConfigurationDefault Configuration Values{"redis":{"host":"localhost","port":6379,"database":0},"safety_threshold":10,"requests_module_name":"requests"}Configuration Keysredis: Provides thehost,portanddatabaseof the
Redis instancesafety_threshold: A rate-limited exception will be raised at(realm_max_requests - safety_threshold). Prevents going over the
limit of services in scenarios where a large amount of requests are
issued in parallelrequests_module_name: Provides the name of theRequestsmodule used in the request lambdas. Should not need to be changed
unless you importRequestsas another name.Overriding Configuration ValuesWithrequests-respectful.config.ymlThe library auto-detects the presence of a YAML file namedrequests-respectful.config.ymlat the root of your project and will
attempt to load configuration values from it.Example:requests-respectful.config.ymlredis:host:0.0.0.0port:6379database:5safety_threshold:25With theconfigure()class methodIf you don’t like having an extra file lying around, the library can
also be configured at runtime using theconfigure()class method.RespectfulRequester.configure(redis={"host":"0.0.0.0","port":6379,"database":5},safety_threshold=25)In both cases, the resulting active configuration would be:RespectfulRequester._config()Out[1]:{"redis":{"host":"0.0.0.0","port":6379,"database":5},"safety_threshold":25,"requests_module_name":"requests"}UsageIn your quest to userequests-respectful, you should only ever have to
bother with one class:RespectfulRequester. Instance this class and
you can perform all important operations.Before each example, it is assumed that the following code has already
been executed.fromrequests_respectfulimportRespectfulRequesterrr=RespectfulRequester()RealmsRealms are simply named containers that are provided with a maximum
requesting rate. You are responsible of the management (i.e. CRUD) of
your realms.Realms track the HTTP requests that are performed under them and will
raise a catchable rate limit exception if you are over their allowed
requesting rate.Fetching the list of Realmsrr.fetch_registered_realms()This returns a list of currently registered realm names.Registering a Realmrr.register_realm("Google",max_requests=10,timespan=1)rr.register_realm("Github",max_requests=100,timespan=60)rr.register_realm("Twitter",max_requests=150,timespan=300)# ORrealm_tuples=[["Google",10,1],["Github",100,60],["Twitter",150,300]]rr.register_realms(realm_tuples)Either of these registers 3 realms: *Googleat a maximum requesting
rate of 10 requests per second *Githubat a maximum requesting rate
of 100 requests per minute *Twitterat a maximum requesting rate of
150 requests per 5 minutesUpdating a Realmrr.update_realm("Google",max_requests=25,timespan=5)This updates the maximum requesting rate ofGoogleto 25 requests per
5 seconds.Getting the maximum requests value of a Realmrr.realm_max_requests("Google")This would return 25.Getting the timespan value of a Realmrr.realm_timespan("Google")This would return 5.Unregistering a Realmrr.unregister_realm("Google")This would unregister theGooglerealm, preventing further queries
from executing on it.Unregistering multiple Realmsrr.unregister_realms(["Google","Github","Twitter"])This would unregister all 3 realms in one operation, preventing further
queries from executing on them.RequestingUsingRequestsHTTP verb methodsThe library supports proxying calls to the 7RequestsHTTP verb
methods (DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT). This is
literally aRequestsmethod so go crazy with yourparams,body,headers,authetc. kwargs. The only major difference is that arealmkwarg is expected. Awaitboolean kwargs can also be provided
(the behavior is explained later).These are all valid calls:rr.get("http://httpbin.org",realms=["HTTPBin"])rr.post('http://httpbin.org/post',data={'key':'value'},realms=["HTTPBin"],wait=True)rr.put('http://httpbin.org/put',data={'key':'value'},realms=["HTTPBin"])rr.delete('http://httpbin.org/delete',realms=["HTTPBin"])If not rate-limited, these would return your usualrequests.Responseobject.Using a request lambaIf you are a purist and prefer not using fancy proxying, you are also
allowed to create a lambda of yourRequestscall and pass it to therequest()instance method.request_func=lambda:requests.post('http://httpbin.org/post',data={'key':'value'})rr.request(request_func,realms=["HTTPBin"],wait=True)If not rate-limited, this would return your usualrequests.Responseobject.Multiple realms per requestStarting in 0.2.0, you can have a single request count against multiple
realms. The kwarg has been changed fromrealmtorealmsand
works as you would expect it to.rr.get("http://httpbin.org",realms=["HTTPBin","HTTPBinUser123","HTTPBinServer3"])The kwargrealmhas been deprecated on requesting instance methods.
It will still work with a warning until 0.3.0Handling exceptionsExecuting these calls will either return arequests.Responseobject
with the results of the HTTP call or raise a
RequestsRespectfulRateLimitedError exception. This means that you’ll
likely want to catch and handle that exception.fromrequests_respectfulimportRequestsRespectfulRateLimitedErrortry:response=rr.get("http://httpbin.org",realm="HTTPBin")exceptRequestsRespectfulRateLimitedError:pass# Possibly requeue that call or wait.ThewaitkwargBoth ways of requesting accept awaitkwarg that defaults to False. If
switched on and the realm is currently rate-limited, the process will
block, wait until it is safe to send requests again and perform the
requests then. Waiting is perfectly fine for scripts or smaller
operations but is discouraged for large, multi-realm, parallel tasks
(i.e. Background Tasks like Celery workers).TestsExist?YesExhaustive?YesFacepalm tactics?Yes -  Redis calls aren't mocked and google.com gets a few friendly callsRun them withpython-mpytest tests--specFAQWhoa, whoa, whoa! Redis?!Yes. The use of Redis allows forrequests-respectfulto go
multi-thread, multi-process and even multi-machine while still
respecting the maximum requesting rates of registered realms. Operations
like Redis’ SETEX are key in designing and working with rate-limiting
systems. If you are doing Python development, there is a decent chance
you already work with Redis as it is one of the two options to use as
Celery’s backend and one of the 2 major caching options in Web
development. If not, you can always keep things clean and use aDocker
Containeror evenbuild it from
source. Redis has kept a
consistent record over the years of being lightweight, solid software.How is this different than other throttling libraries?Most other libraries will ask you to specify an interval at which to
send requests and will literally loop overrequest()...time.sleep(interval). This one will allow to send as
many as you want, as fast as you want, as long as you are under the
maximum requesting rate of your realm.Other libraries don’t have the concept of realms and separate
requesting rate rules.Other libraries don’t scale outside of the process.Most other libraries don’t integrate this neatly withRequestsRoadmap / Contribution IdeasProvide some introspection methods to get live realm statsCreate a curses realm stats monitorProvide real-life use casesRead the Docs RST DocumentationMock out the Redis calls in the testsMock out the Requests calls in the tests