Package: async-http-requests
Async RequestsPython library to handle asynchronous http requests, built on top of therequestslibraryInstallationUsageInstallationPyPI pagepipinstallasync-http-requestsUsageThe library provide support for asynchronous http requests, using the consumer-producer pattern leveraging the built-in python modu.Instantiate the classAsyncHTTPspecifying a list ofRequestObject, (you can specifyN_PRODUCERSandN_CONSUMERS: default values are10for both)TheRequestObjectsupports all keyword arguments of the requests methods (headers,params,data, ...). It allows you to specify different keyword arguments across different requestsfromAysncRequestsimportAsyncHTPP,RequestObject,RequestType,# Public API endpoint, it retrieves all public APIs listed under params specification# Refer to this if you want to know more https://api.publicapis.org/api='https://api.publicapis.org/entries'# default N_PRODUCERS and N_CONSUMERS to 10endpoints=[RequestObject(url=api,params={"title":"cat"}),RequestObject(url=api,params={"title":"dog"})]requests=AsyncHTTP(url=endpoints)# specify number of producers and consumersrequests=AsyncHTTP(url=endpoints,N_PRODUCERS=10,N_CONSUMERS=10)To get the responses you can either call the genericasync_requestmethod explicitly specifying the http request type by passing to therequest_typeargument aRequestTypeEnum (GET,POST,PUT,PATCH,DELETE,HEAD),
or you can use theasync_get,async_post,async_delete,async_put,async_patch,async_headmethod without having to spepcify the request type.requests.async_request(request_type=RequestType.GET,)requests.async_get()All methods support additional fixed keyword arguments such asheaders,authetc. as per the usual requests module, in case you need certain arguments to stay fixed across requestsFor the additional paramaters refer to the requests module documentationRequests docsheaders={'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36'}# Keyword arguments FIXED for all requests# Specify the RequestTyperequests.async_request(request_type=RequestType.GET,haeders=headers# using the same header across all the requests)# Using async_getrequests.async_get(headers=headers)# using the same header across all the requestsAll methods support the use of callback functions, to be used by the consumers on therequest.Responseobject generated by the producers, as they become available in theasyncio.Queue.In case of unsuccessfull requests the parametermax_retriescan be specified (default is 0).
Whenmax retriesis set larger than zero, any http request that has returned an error will be sent again up to the specified number of times if not successfull.
If the given requests still generates problems after all the attempts it will be pushed to therequests.error_responseand won't be found inrequests.responseThemax_retriesparamater can be useful when you are sending a big number of requests to the same server, in which case you can get an error just momentarily even if nothing is really wrong.defexample_callaback(response:request.Response):returnresponse.status_coderequests.async_request(request_type=RequestType.GET,max_retries=5,# retry sendign the requests 5 times for all the unsuccessfull onescallback=example_callback,headers=headers)# no retries specified -> defaults to no further attempts in case of errorsrequests.async_get(headers=headers,callback=lambdax:x.json()# lambda as callback)The results will be stored in a list object, where you'll find either therequests.Responseobjects, or the output of thecallbackfunction.Requests that return an error code will be saved inrequests.error_responseA summary of all the requests that were not successfull (if any) can be displayed summarized in apandas.DataFrameobject through therequests.error_dataattributerequests.responserequests.error_responserequests.error_dataCheck thetest.pyscript for an example.