Package: fastget
FastGETEasy parallel and concurrent GET requestsThe idea of this package is to wrapmultiprocessingandasyncconcurrency and allow the user to perform thousands of requests in parallel and concurrently without having to worry about pools of processes and async event loops.It only supports GET requests. More methods could be implemented but this initial version is as simple as it. Also implementing other methods will break the beauty of the name.The input is an iterable and the output is a generator. As soon as the requests get answer they are yielded until all the requests of the input are done.Each element of the iterable is a tuple withIdandURL.
Each element yielded is a tuple with the sameIdand the json response. With theIdyou can
later map the responses.This is useful for cases where you have a huge amount of requests to perform to an API and you
need to do them as fast as possible.The client by default detects the number of CPUs available and starts one process per each CPU.
Then chunks the input iterator to provide requests to all the processes.
Each process for each task opens an event loop and performs all those requests concurrently. Once
all requests are awaited, the chunk with all the responses is returned back to the main process.
This is why we can see that our generator is receiving the responses un bulks.Install:From PyPi:pip install fastgetUsage:Use always context manager:>>>fromfastgetimportFastGET>>>withFastGET()asclient:...responses=client.get([(1,"http://localhost:12345"),(2,"http://localhost:12345")])...forresponseinresponses:...print(response)...fastgetINFOStartprocessingrequestswithFastGETparameters:fastgetINFOnum_workers:8fastgetINFOsingle_submit_size:5000fastgetINFOpool_submit_size:50000fastgetINFOqueue_max_size:100000(2,{'message':'Hello World!'})(1,{'message':'Hello World!'})fastgetINFOAllrequestsprocessed:fastgetINFOTotalrequests:2fastgetINFOTotaltime:0.05fastgetINFORequests/s:34.16You can provide a generator to don't blow up the memory:>>>fromfastgetimportFastGET>>>fromcollectionsimportdeque>>>>>>defmygen():...foriinrange(100_000):...yield(i,"http://localhost:12345")...>>>withFastGET()asclient:...responses=client.get(mygen())..._=deque(responses)...fastgetINFOStartprocessingrequestswithFastGETparameters:fastgetINFOnum_workers:8fastgetINFOqueue_max_size:100000fastgetINFOinput_chunk_size:10000fastgetINFOpool_submit_size:1000fastgetINFOAllrequestsprocessed:fastgetINFOTotalrequests:100000fastgetINFOTotaltime(s):36.35fastgetINFORequests/s:2750.93ParametersYou can configure some parameters like the amount of workers or how the client chunks the input:fastget.FastGETparameters:num_workers:type: intdefault: os.cpu_count()description: Number of processes to open with multiprocessingqueue_max_size:type: intdefault: 100.000description: Maximum number of items that can be enqueued. This default number proved to not blow up the memory and to have enough items in the queue to have always work to do with 8 processes. Feel free to adjust it, just watch out the memory usage.input_chunk_size:type: intdefault: 10.000description: This is the size of the chunks for the input. We will be reading the input iterator in chunks of this size up toqueue_max_size.pool_submit_size:type: intdefault: 1.000description: Each chunk ofinput_chunk_sizewill also be chunked to minor chunks of this size before being submited to the pool. The workers will be consuming chunks of this size and each of these chunks will be requested in an event loop.fastget.FastGET.getParameters:ids_and_urls:type: Iterable[Tuple[int, str]]required: Truedescription: Provide the tuples containing the ID of the request and the URL to be requested.Response: Generator[Tuple[int, str], None, None]. For each input tuple an output tuple will be returned containing the same ID + the json of the response.