Package: torequests
torequestsBriefly speaking, requests & aiohttp wrapper for asynchronous programming rookie, to shorten the code quantity.Install:pip install torequests -Urequirementspython3.7+ (follow requests settings)| requests			# python
| aiohttp >= 3.6.2 	# python3FeaturesInspired bytomorrow, to make async-coding brief & smooth, compatible for win32 / python 2&3.Convert any funtions into async-mode with concurrent.futures.Wrap requests lib with concurrent.futures to enjoy the concurrent performance.Simplify aiohttp, make itrequests-like.Add FailureException to check the context of request failure.Add frequency control, prevent from anti-spider based on frequency check.Add retry for request. (supportresponse_validator)Alenty of common crawler utils.Compatible withuniparserGetting started1. Async, threads - make functions asynchronousfromtorequestsimportthreads,Asyncimporttime@threads(5)deftest1(n):time.sleep(n)return'test1 ok'deftest2(n):time.sleep(n)return'test1 ok'start=int(time.time())# here async_test2 is same as test1async_test2=Async(test2)future=test1(1)# future run in non blocking thread poolprint(future,',%ss passed'%(int(time.time()-start)))# call future.x will block main thread and get the future.result()print(future.x,',%ss passed'%(int(time.time()-start)))# output:# <NewFuture at 0x34b1d30 state=running> , 0 s passed# test1 ok , 1 s passed2. tPool - thread pool for async-requestsfromtorequests.mainimporttPoolfromtorequests.logsimportprint_infofromtorequests.utilsimportttimereq=tPool()test_url='http://p.3.cn'defcallback(task):return(len(task.content),print_info(ttime(task.task_start_time),'-> ',ttime(task.task_end_time),',',round(task.task_cost_time*1000),'ms'))ss=[req.get(test_url,retry=2,callback=callback)foriinrange(3)]# or [i.x for i in ss]req.x# task.cx returns the callback_resultss=[i.cxforiinss]print_info(ss)# [2020-01-21 16:56:23] temp_code.py(11): 2020-01-21 16:56:23 ->  2020-01-21 16:56:23 , 54 ms# [2020-01-21 16:56:23] temp_code.py(11): 2020-01-21 16:56:23 ->  2020-01-21 16:56:23 , 55 ms# [2020-01-21 16:56:23] temp_code.py(11): 2020-01-21 16:56:23 ->  2020-01-21 16:56:23 , 57 ms# [2020-01-21 16:56:23] temp_code.py(18): [(612, None), (612, None), (612, None)]2.1 Performance.[3.7.1(v3.7.1:260ec2c36a,Oct202018,14:57:15)[MSCv.191564bit(AMD64)]]:2000/2000,100.0%,cost4.2121seconds,475.0qps.importtimeitfromtorequestsimporttPoolimportsysreq=tPool()start_time=timeit.default_timer()oks=0total=2000# concurrent all the taskstasks=[req.get('http://127.0.0.1:8080')fornuminrange(total)]fortaskintasks:r=task.xifr.text=='ok':oks+=1end_time=timeit.default_timer()succ_rate=oks*100/totalcost_time=round(end_time-start_time,4)version=sys.versionqps=round(total/cost_time,0)print('[{version}]:{oks}/{total},{succ_rate}%, cost{cost_time}seconds,{qps}qps.'.format(**vars()))3. Requests - aiohttp-wrapperfromtorequests.dummyimportRequestsfromtorequests.logsimportprint_infofromtorequests.utilsimportttimereq=Requests(frequencies={'p.3.cn':(2,1)})defcallback(task):return(len(task.content),print_info(ttime(task.task_start_time),'->',ttime(task.task_end_time),',',round(task.task_cost_time*1000),'ms'))ss=[req.get('http://p.3.cn',retry=1,timeout=5,callback=callback)foriinrange(4)]req.x# this line can be removedss=[i.cxforiinss]print_info(ss)# [2020-01-21 18:15:33] temp_code.py(11): 2020-01-21 18:15:32 -> 2020-01-21 18:15:33 , 1060 ms# [2020-01-21 18:15:33] temp_code.py(11): 2020-01-21 18:15:32 -> 2020-01-21 18:15:33 , 1061 ms# [2020-01-21 18:15:34] temp_code.py(11): 2020-01-21 18:15:32 -> 2020-01-21 18:15:34 , 2081 ms# [2020-01-21 18:15:34] temp_code.py(11): 2020-01-21 18:15:32 -> 2020-01-21 18:15:34 , 2081 ms# [2020-01-21 18:15:34] temp_code.py(20): [(612, None), (612, None), (612, None), (612, None)]3.1 Performance.aiohttp is almostly 3 times faster than requests + ThreadPoolExecutor, even without uvloop on windows10.3.7.1(v3.7.1:260ec2c36a,Oct202018,14:57:15)[MSCv.191564bit(AMD64)]sync_test:2000/2000,100.0%,cost1.2965seconds,1543.0qps.async_test:2000/2000,100.0%,cost1.2834seconds,1558.0qps.Syncusagehaslittleperformancelost.importasyncioimportsysimporttimeitfromtorequests.dummyimportRequestsdefsync_test():withRequests()asreq:start_time=timeit.default_timer()oks=0total=2000# concurrent all the taskstasks=[req.get('http://127.0.0.1:8080')fornuminrange(total)]req.xfortaskintasks:iftask.text=='ok':oks+=1end_time=timeit.default_timer()succ_rate=oks*100/totalcost_time=round(end_time-start_time,4)qps=round(total/cost_time,0)print(f'sync_test:{oks}/{total},{succ_rate}%, cost{cost_time}seconds,{qps}qps.')asyncdefasync_test():req=Requests()asyncwithRequests()asreq:start_time=timeit.default_timer()oks=0total=2000# concurrent all the taskstasks=[req.get('http://127.0.0.1:8080')fornuminrange(total)]fortaskintasks:r=awaittaskifr.text=='ok':oks+=1end_time=timeit.default_timer()succ_rate=oks*100/totalcost_time=round(end_time-start_time,4)qps=round(total/cost_time,0)print(f'async_test:{oks}/{total},{succ_rate}%, cost{cost_time}seconds,{qps}qps.')if__name__=="__main__":print(sys.version)sync_test()loop=asyncio.get_event_loop()loop.run_until_complete(async_test())3.2 using torequests.dummy.Requests in async environment.importasyncioimportuvicornfromstarlette.applicationsimportStarlettefromstarlette.responsesimportPlainTextResponsefromtorequests.dummyimportRequestsapi=Starlette()api.req=Requests()@api.route('/')asyncdefindex(req):# await for Response or FailureExceptionr=awaitapi.req.get('http://p.3.cn',timeout=(1,1))returnPlainTextResponse(r.content)if__name__=="__main__":uvicorn.run(api)3.3 using torequests.aiohttp_dummy.Requests for better performance.Removes the frequency_controller & sync usage (task.x) & compatible args of requests for good performance, but remains retry / callback / referer_info.# -*- coding: utf-8 -*-fromtorequests.aiohttp_dummyimportRequestsfromasyncioimportget_event_loopasyncdefmain():url='http://httpbin.org/json'# simple usage, catch_exception defaults to TrueasyncwithRequests()asreq:r=awaitreq.get(url,retry=1,encoding='u8',timeout=3)print(r.ok)print(r.url)print(r.content[:5])print(r.status_code)print(r.text[:5])print(r.json()['slideshow']['title'])print(r.is_redirect)print(r.encoding)# or use `req = Requests()` without contextreq=Requests()r=awaitreq.get(url,referer_info=123,callback=lambdar:r.referer_info+1,)print(r)# 124get_event_loop().run_until_complete(main())4. utils: some useful crawler toolkitsClipboardWatcher: watch your clipboard changing.
Counts: counter while every time being called.
Null: will return self when be called, and alway be False.
Regex: Regex Mapper for string -> regex -> object.
Saver: simple object persistent toolkit with pickle/json.
Timer: timing tool.
UA: some common User-Agents for crawler.
curlparse: translate curl-string into dict of request.
md5: str(obj) -> md5_string.
print_mem: show the proc-mem-cost with psutil, use this only for lazinesssss.
ptime: %Y-%m-%d %H:%M:%S -> timestamp.
ttime: timestamp -> %Y-%m-%d %H:%M:%S
slice_by_size: slice a sequence into chunks, return as a generation of chunks with size.
slice_into_pieces: slice a sequence into n pieces, return a generation of n pieces.
timeago: show the seconds as human-readable.
unique: unique one sequence.
find_one: use regex like Javascript to find one string with index(like [0], [1]).
find_jsons: find all json substrings from a mass string contains list / dict JSON.
...BenchmarksBenchmark of concurrent is not very necessary and accurate, just take a look.Test Server: golang net/httpSource code:go_test_server.goClient Testing CodeSource code:py_test_client.pyTest Result on Windows (without uvloop)Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz (12 CPUs), ~2.2GHz; 12 logical CPUs.Windows-10-10.0.18362-SP03.7.1(v3.7.1:260ec2c36a,Oct202018,14:57:15)[MSCv.191564bit(AMD64)]['aiohttp(3.6.2)','torequests(4.9.14)','requests(2.23.0)','httpx(0.12.1)']================================================================================test_aiohttp:2000/2000=100.0%,cost1.289s,1551qps,100%standard.test_dummy:2000/2000=100.0%,cost1.397s,1432qps,92%standard.test_aiohttp_dummy:2000/2000=100.0%,cost1.341s,1491qps,96%standard.test_httpx:2000/2000=100.0%,cost4.065s,492qps,32%standard.test_tPool:2000/2000=100.0%,cost5.161s,388qps,25%standard.Test Result on Linux (with uvloop)Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz; 1 logical CPU.Linux-4.15.0-13-generic-x86_64-with-Ubuntu-18.04-bionic3.7.3(default,Apr32019,19:16:38)[GCC8.0.120180414(experimental)[trunkrevision259383]]['aiohttp(3.6.2)','torequests(4.9.14)','requests(2.23.0)','httpx(0.12.1)']================================================================================test_aiohttp:2000/2000=100.0%,cost0.652s,3068qps,100%standard.test_dummy:2000/2000=100.0%,cost0.802s,2495qps,81%standard.test_aiohttp_dummy:2000/2000=100.0%,cost0.726s,2754qps,90%standard.test_httpx:2000/2000=100.0%,cost2.349s,852qps,28%standard.test_tPool:2000/2000=100.0%,cost2.708s,739qps,24%standard.Conclusionaiohttpis the fastest, for the cython utilsaiohttp's qps is 3068 on 1 cpu linux with uvloop, near to golang's 3300.torequests.dummy.Requestsbased onaiohttp.about8%performance lostwithoutuvloop.about20%performance lost with uvloop.torequests.aiohttp_dummy.Requestsbased onaiohttp.less than4%performance lostwithoutuvloop.less than10%performance lost with uvloop.httpxis faster thanrequests + threading,but not very obviously on linux.PS:golang - net/http's performance`2000 / 2000, 100.00 %, cost 0.26 seconds, 7567.38 qps` on windows (12 cpus)
`2000 / 2000, 100.00 %, cost 0.61 seconds, 3302.48 qps` on linux (1 cpu)
   1. slower than windows, because golang benefit from multiple CPU count
   2. linux 1 cpu, but windows is 12golang http client testing code:go_test_client.goDocumentationDocument & UsageLicenseMIT license