Package: request-boost
request-boostA better requests and urllib package.A simple package for hitting multiple URLs and performing GET/POST requests in parallel.Setuppipinstallrequest-boostImportant:Virtual Environment is recommendedUsage# Sample datanumber_of_sample_urls=1000urls=[f'https://postman-echo.com/get?random_data={test_no}'fortest_noinrange(number_of_sample_urls)]headers=[{'sample_header':test_no}fortest_noinrange(number_of_sample_urls)]fromrequest_boostimportboosted_requestsresults=boosted_requests(urls=urls)print(results)fromrequest_boostimportboosted_requestsresults=boosted_requests(urls=urls,no_workers=16,max_tries=5,timeout=5,headers=headers,verbose=True,parse_json=True)print(results)fromrequest_boostimportboosted_requests# Confignumber_of_sample_urls=100# Sample dataurls=[f'https://postman-echo.com/get?random_data={test_no}'fortest_noinrange(number_of_sample_urls)]post_urls=[f'https://postman-echo.com/post'fortest_noinrange(number_of_sample_urls)]headers=[{'sample_header':test_no}fortest_noinrange(number_of_sample_urls)]data=[{'sample_data':test_no}fortest_noinrange(number_of_sample_urls)]# Required for POST requests,#For POST request data can be just list of empty dict but not NONEsimple_results=boosted_requests(urls=urls,no_workers=16,max_tries=5,timeout=5,headers=None,verbose=False,parse_json=True)header_results=boosted_requests(urls=urls,no_workers=16,max_tries=5,timeout=5,headers=headers,parse_json=True)post_results=boosted_requests(urls=post_urls,no_workers=16,max_tries=5,timeout=5,headers=headers,data=data,verbose=True,parse_json=True)Documentationboosted_requests(urls,no_workers=32,max_tries=5,after_max_tries="assert",timeout=10,headers=None,data=None,verbose=True,parse_json=True,)GetdatafromAPIsinparallelbycreatingworkersthatprocessinthebackground:paramurls:listofURLS:paramno_workers:maximumnumberofparallelprocesses{Default::32}:parammax_tries:MaximumnumberoftriesbeforefailingforaspecificURL{Default::5}:paramafter_max_tries:Whattodoifnotsuccessfullafter"max_tries"foraspecificURL,oneof{"assert","break"}{Default::assert}:paramtimeout:Waitingtimeperrequest{Default::10}:paramheaders:HeadersifanyfortheURLrequests:paramdata:dataifanyfortheURLrequests(WherevernotNoneaPOSTrequestismade):paramverbose:Showprogress[TrueorFalse]{Default::True}:paramparse_json:Parseresponsetojson[TrueorFalse]{Default::True}:return:ListofresponseforeachAPI(orderismaintained)CreditsMaintained byKuldeep Singh SidhuGithub:github/singhsidhukuldeephttps://github.com/singhsidhukuldeepWebsite:Kuldeep Singh Sidhu (Website)http://kuldeepsinghsidhu.comLinkedIn:Kuldeep Singh Sidhu (LinkedIn)https://www.linkedin.com/in/singhsidhukuldeep/ContributorsThe full list of all the contributors is availablehereTODO:: Want to contribute?Make the backend used (urllib/request library) an option that the user can choose, by default we use urllibFor parallel processing add options for multi-processing along with multi-threadingSet-up tests for edge cases and changes verificationSet-up CI/CD pipleine (possibly using GitHub actions) to publish changes to PyPiImproeve the doc-strings documentation to add more explanantion and examplesAdd a message queue to deploy the service across machinesAdd option to run URL requests with Self signed certificateverify=FalseAdd option to supress warningsAdd progess bars from tqdm and asciiAdd option to add session and authSay ThanksIf this helped you in any way, it would be great if you could share it with others.Steps for publishing topypi[This is just for me, Maybe!]pip3 install setuptools twineGo to project folderpython3 setup.py sdisttwine upload --repository-url https://upload.pypi.org/legacy/ dist/*ORGo to your project folder and:pip3installsetuptoolstwine

python3setup.pysdist
twineupload--repository-urlhttps://upload.pypi.org/legacy/dist/*Current StatusCHANGE LOGLATEST0.8 (06/06/2023)added a new :paramafter_max_tries: What to do if not successfull after"max_tries"for a specific URL,
one of {"assert","break"} {Default::assert}PAST0.1 (16/09/2021)Released first version(stable) of request-boost.0.2 (16/09/2021)Fixed header issueAdded POST functionality0.3.1 (17/09/2021)Added Verbose0.4 (17/09/2021)Added Verbose timingImproved performance0.5 (16/10/2021)Added Encoding fixImproved Readability0.6 (29/12/2021)Added option to not parse the JSONImproved documentation0.7 (29/12/2021)Fixed timeout key error issue, to returnNonefor timeoutsImproved documentationRead more athttps://github.com/singhsidhukuldeep/request-boost