Saravanan666/pypi-org huggingface dataset:
Package: aioscpy
AioscpyAn asyncio + aiolibs crawler  imitate scrapy frameworkEnglish |中文OverviewAioscpy framework is base on opensource project Scrapy & scrapy_redis.Aioscpy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages.Dynamic variable injection is implemented and asynchronous coroutine feature support.Distributed crawling/scraping.RequirementsPython 3.8+Works on Linux, Windows, macOS, BSDInstallThe quick way:# defaultpipinstallaioscpy# at latest versionpipinstallgit+https://github.com/ihandmine/aioscpy# install all dependenciespipinstallaioscpy[all]# install extra packagespipinstallaioscpy[aiohttp,httpx]Usagecreate project spider:aioscpystartprojectproject_quotescd project_quotes
aioscpy genspider quotesquotes.py:fromaioscpy.spiderimportSpiderclassQuotesSpider(Spider):name='quotes'custom_settings={"SPIDER_IDLE":False}start_urls=['https://quotes.toscrape.com/tag/humor/',]asyncdefparse(self,response):forquoteinresponse.css('div.quote'):yield{'author':quote.xpath('span/small/text()').get(),'text':quote.css('span.text::text').get(),}next_page=response.css('li.next a::attr("href")').get()ifnext_pageisnotNone:yieldresponse.follow(next_page,self.parse)create single script spider:aioscpyonespidersingle_quotessingle_quotes.py:fromaioscpy.spiderimportSpiderfromanti_headerimportHeaderfrompprintimportpprint,pformatclassSingleQuotesSpider(Spider):name='single_quotes'custom_settings={"SPIDER_IDLE":False}start_urls=['https://quotes.toscrape.com/',]asyncdefprocess_request(self,request):request.headers=Header(url=request.url,platform='windows',connection=True).randomreturnrequestasyncdefprocess_response(self,request,response):ifresponse.statusin[404,503]:returnrequestreturnresponseasyncdefprocess_exception(self,request,exc):raiseexcasyncdefparse(self,response):forquoteinresponse.css('div.quote'):yield{'author':quote.xpath('span/small/text()').get(),'text':quote.css('span.text::text').get(),}next_page=response.css('li.next a::attr("href")').get()ifnext_pageisnotNone:yieldresponse.follow(next_page,callback=self.parse)asyncdefprocess_item(self,item):self.logger.info("{item}",**{'item':pformat(item)})if__name__=='__main__':quotes=SingleQuotesSpider()quotes.start()run the spider:aioscpycrawlquotes
aioscpyrunspiderquotes.pystart.py:fromaioscpy.crawlerimportcall_grace_instancefromaioscpy.utils.toolsimportget_project_settings"""start spider method one:from cegex.baidu import BaiduSpiderfrom cegex.httpbin import HttpBinSpiderprocess = CrawlerProcess()process.crawl(HttpBinSpider)process.crawl(BaiduSpider)process.start()"""defload_file_to_execute():process=call_grace_instance("crawler_process",get_project_settings())process.load_spider(path='./cegex',spider_like='baidu')process.start()defload_name_to_execute():process=call_grace_instance("crawler_process",get_project_settings())process.crawl('baidu',path="./cegex")process.start()if__name__=='__main__':load_file_to_execute()more commands:aioscpy-hReadyplease submit your sugguestion to owner by issueThanksaiohttpscrapyloguruhttpx
Package: aioscrape
A scraping library on top ofaiohttpandparsechain. Note that this isalphasoftware.Installationpip install aioscrapeUsagefromaioscrapeimportrun,fetch,settingsfromaioscrape.middlewareimportlast_fetch,make_filecachefromaioscrape.utilsimportSOME_HEADERS# To not look like a botfromurllib.parseimporturljoinfromparsechainimportCfromfuncyimportlcat,lconcatdefmain():# Settings are scoped and can be redefined later with another "with"cache=make_filecache('.fcache')withsettings(headers=SOME_HEADERS,middleware=[cache,last_fetch]):print(run(scrape_all()))asyncdefscrape_all():# All the settings in scope like headers and middleware are applied to fetch()start_page=awaitfetch(START_URL)# AioScrape integrates with parsechain to make extracting a breezeurls=start_page.css('.pagingLinks a').attrs('href')list_urls=[urljoin(start_page.url,page_url)forpage_urlinurls]# Using asyncio.wait() and friends to run requests in parallellist_pages=[start_page]+awaitwait_all(map(fetch,list_urls))# Scrape articlesresult=lcat(awaitwait_all(map(scrape_articles,list_pages)))write_to_csv('export.csv',result)asyncdefscrape_articles(list_page):urls=list_page.css('#headlines .titleLink').attrs('href')abs_urls=[urljoin(list_page.url,url)forurlinurls]returnawaitwait_all(map(scrape_article,abs_urls))asyncdefscrape_article(url):resp=awaitfetch(url)returnresp.root.multi({'url':C.const(resp.url),'title':C.microdata('headline').first,'date':C.microdata('datePublished').first,'text':C.microdata('articleBody').first,'contacts':C.css('.sidebars .contact p').map(C.inner_html+html_to_text)+lconcat+''.join,})if__name__=='__main__':main()TODOResponse.follow()non-GET requestswork with forms
Package: aioscrapy
Python async library for web scrapingInstallingpip install aioscrapyUsagePlain text scrapingimportasyncioimportjsonfromaioscrapyimportClient,WebTextClient,SingleSessionPool,Dispatcher,SimpleWorkerclassCustomClient(Client[str,dict]):def__init__(self,client:WebTextClient):self._client=clientasyncdeffetch(self,key:str)->dict:data=awaitself._client.fetch(key)returnjson.loads(data)asyncdefmain():pool=SingleSessionPool()dispatcher=Dispatcher(['https://httpbin.org/get'])client=CustomClient(WebTextClient(pool))worker=SimpleWorker(dispatcher,client)result=awaitworker.run()returnresultloop=asyncio.get_event_loop()print(loop.run_until_complete(main()))Byte content downloadingimportasynciofromaioscrapyimportClient,WebByteClient,SingleSessionPool,Dispatcher,SimpleWorkerclassCustomClient(Client[str,bytes]):def__init__(self,client:WebByteClient):self._client=clientasyncdeffetch(self,key:str)->bytes:data=awaitself._client.fetch(key)returndataasyncdefmain():pool=SingleSessionPool()dispatcher=Dispatcher(['https://httpbin.org/image'])client=CustomClient(WebByteClient(pool))worker=SimpleWorker(dispatcher,client)result=awaitworker.run()returnresultloop=asyncio.get_event_loop()data:dict=loop.run_until_complete(main())forurl,byte_contentindata.items():print(url+": "+str(len(byte_content))+" bytes")
Package: aio-scrapy
aio-scrapyAn asyncio + aiolibs crawler  imitate scrapy frameworkEnglish |中文Overviewaio-scrapy framework is base on opensource project Scrapy & scrapy_redis.aio-scrapy implements compatibility with scrapyd.aio-scrapy implements redis queue and rabbitmq queue.aio-scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages.Distributed crawling/scraping.RequirementsPython 3.9+Works on Linux, Windows, macOS, BSDInstallThe quick way:# Install the latest aio-scrapypipinstallgit+https://github.com/conlin-huang/aio-scrapy# defaultpipinstallaio-scrapy# Install all dependenciespipinstallaio-scrapy[all]# When you need to use mysql/httpx/rabbitmq/mongopipinstallaio-scrapy[aiomysql,httpx,aio-pika,mongo]Usagecreate project spider:aioscrapystartprojectproject_quotescd project_quotes
aioscrapy genspider quotesquotes.pyfromaioscrapy.spidersimportSpiderclassQuotesMemorySpider(Spider):name='QuotesMemorySpider'start_urls=['https://quotes.toscrape.com']asyncdefparse(self,response):forquoteinresponse.css('div.quote'):yield{'author':quote.xpath('span/small/text()').get(),'text':quote.css('span.text::text').get(),}next_page=response.css('li.next a::attr("href")').get()ifnext_pageisnotNone:yieldresponse.follow(next_page,self.parse)if__name__=='__main__':QuotesMemorySpider.start()run the spider:aioscrapycrawlquotescreate single script spider:aioscrapygenspidersingle_quotes-tsinglesingle_quotes.py:fromaioscrapy.spidersimportSpiderclassQuotesMemorySpider(Spider):name='QuotesMemorySpider'custom_settings={"USER_AGENT":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36",'CLOSE_SPIDER_ON_IDLE':True,# 'DOWNLOAD_DELAY': 3,# 'RANDOMIZE_DOWNLOAD_DELAY': True,# 'CONCURRENT_REQUESTS': 1,# 'LOG_LEVEL': 'INFO'}start_urls=['https://quotes.toscrape.com']@staticmethodasyncdefprocess_request(request,spider):""" request middleware """pass@staticmethodasyncdefprocess_response(request,response,spider):""" response middleware """returnresponse@staticmethodasyncdefprocess_exception(request,exception,spider):""" exception middleware """passasyncdefparse(self,response):forquoteinresponse.css('div.quote'):yield{'author':quote.xpath('span/small/text()').get(),'text':quote.css('span.text::text').get(),}next_page=response.css('li.next a::attr("href")').get()ifnext_pageisnotNone:yieldresponse.follow(next_page,self.parse)asyncdefprocess_item(self,item):print(item)if__name__=='__main__':QuotesMemorySpider.start()run the spider:aioscrapyrunspiderquotes.pymore commands:aioscrapy-hmore exampleDocumentationdocReadyplease submit your sugguestion to owner by issueThanksaiohttpscrapy
Package: aioscrapy-redis
Python async library for web scraping
PyPI version License: MITBuild Status codecov codebeat badge Codacy BadgeInstalling
pip install aioscrapy_redis
Usage
Plain text scrapingfrom aioscrapy_redis.core.spider import Spiderfrom aioscrapy_redis.https.request import Requestimport refrom urllib.parse import unquote"""
The start url can be placed in start_urls or written to the redis queue
"""class Async_Spider(Spider):name = 'aioscrapy_spider'

redis_key = 'aioscrapy_spider:url'

start_urls = []

def parse(self, response):
    pass
Package: aioscript
info:Base asyncio script with support of threading and multiprocessingInstallationpipinstallaioscriptUsagefromaiohttpimportClientSession,webfromaioscriptimportAbstractScriptclassScript(AbstractScript):defsetup(self):self.session=ClientSession(loop=self.loop)asyncdefclose(self):awaitself.session.close()asyncdefhandle(self,url):asyncwithself.session.get(url)asresponse:ifresponse.status==web.HTTPOk.status_code:print(response.url,'Ok')else:print(response.url,'Not ok')asyncdefpopulate(self):urls=['https://www.python.org/','https://www.python.org/doc/','https://docs.python.org/3/','https://docs.python.org/3/library/concurrency.html','https://docs.python.org/3/library/asyncio.html','https://docs.python.org/3/library/asyncio-eventloop.html',]forurlinurls:yieldurlif__name__=='__main__':Script().run()pythonscript.py--coroutines=10Python 3.6+ is requiredThanksThe library was donated byOcean S.A.Thanks to the company for contribution.
Package: aioscryfall
aioscryfall is a Python client for the Scryfall API. It is primarily designed for asynchronous use with asyncio, but a synchronous client is also provided.The development is essentially feature complete but the documentation is largely absent, at present. (I need to remind myself how mkdocs works and figure out what I want to say)As of right now, Python 3.11 is required because I wanted to use a number of typing features that were added in 3.11. I may change this in the future, but it is a lower priority than getting the API to a point where I am happy with it.Everything is extensively typed and I have tried to make the API as intuitive as possible. I have also tried to make the API as close to the Scryfall API as possible, but I have made some changes to make it more Pythonic (namely converting paginated lists into iterable generators).A minimal use case to get yourself started:importasyncioimportaiohttpfromaioscryfall.api.cardsimportUniqueModefromaioscryfall.clientimportScryfallClientasyncdefget_bolt():asyncwithaiohttp.ClientSession()assession:client=ScryfallClient(session)bolts=[casyncforcinclient.cards.search("lightning bolt",unique=UniqueMode.PRINTS)]returnboltsbolts=asyncio.run(get_bolt())print(len(bolts),bolts[0])Or if you prefer not to use async:fromaioscryfall.api.cardsimportUniqueModefromaioscryfall.sync.clientimportScryfallSyncClientclient=ScryfallSyncClient()bolts=list(client.cards.search("lightning bolt",unique=UniqueMode.PRINTS))print(len(bolts),bolts[0])
Package: aiosearchads
AiosearchadsAsynchronous apple searchads framework for asyncio and PythonDocumentclick meInstallationpip install aiosearchadsSimple usesimportasynciofromaiosearchadsimportAioSearchAdsasyncdeffind_campaigns():core=AioSearchAds(org_id='',token='')data=awaitcore.find_campaigns(0,100)print(data)loop=asyncio.get_event_loop()loop.run_until_complete(find_campaigns())
Package: aioseaweedfs
aioseaweedfsThis is an asyncio client forseaweedfs.Installationpip install aioseaweedfsBasic Usageimportaioseaweedfsasyncdefmain():master=aioseaweedfs.Master()volume,file_ids=awaitmaster.get_assign_key()awaitvolume.post(file_ids[0],"File Content")content=awaitvolume.get(file_ids[0])Filer Usagefiler=aioseaweedfs.Filer()awaitfiler.post("/some/path/to/file.txt","file contents",content_type="text/plain")contents=awaitfiler.get("/some/path/to/file.txt")# contents will always be bytesSeeDocumentation
Package: aioseinfeld
aioseinfeldWhat's the deal with asyncio?Installaioseinfeld requires Python 3.7 or newer. It can be installed from PyPI:$pipinstallaioseinfeldaioseinfeld also depends on having a copy of the script database generated usingscripts by Colin Pollick.
You can build your own copy using those scripts, or download a prebuilt copy with
the following command:$wgethttps://noswap.com/pub/seinfeld.dbUsageaioseinfeld uses context managers to wrap the underlying sqlite database connection.
Create theSeinfeldobject by passing the path to theseinfeld.dbdatabase:fromaioseinfeldimportSeinfeldasyncwithSeinfeld(db_path)asseinfeld:...Get information on individual episodes or seasons:asyncwithSeinfeld(db_path)asseinfeld:season=awaitseinfeld.season(1)episodes=awaitseason.episodesepisodes[0].title# "Good News, Bad News"episodes[0].writers# ["Jerry Seinfeld"]episodes[0].date# date(1990, 6, 14)Quotes can be retrieved by unique ID:asyncwithSeinfeld(db_path)asseinfeld:quote=awaitseinfeld.quote(34665)quote.speaker.name# "George"quote.episode.title# "The Pitch"quote.text# "The show is about nothing."Quotes can also be found by searching:asyncwithSeinfeld(db_path)asseinfeld:awaitseinfeld.search(speaker="Jerry",subject="keys")# [Quote(...), ...]You can even get random quotes with optional search parameters:asyncwithSeinfeld(db_path)asseinfeld:awaitseinfeld.random()# Quote(...)awaitseinfeld.random(subject="parking")# Quote(...)If you want more context around a quote, passages help:asyncwithSeinfeld(db_path)asseinfeld:quote=awaitseinfeld.random()passage=awaitseinfeld.passage(quote,length=5)passage.quotes# [Quote(...), ...]Licenseaiomultiprocess is copyrightJohn Reese, and licensed under
the MIT license.  I am providing code in this repository to you under an open
source license.  This is my personal repository; the license you receive to
my code is from me and not from my employer. See theLICENSEfile for details.
Package: aio_send_dir
aio_send_dirgithubДля чегоНапример, надо отправить coverage отчет после прогона pytestdefpytest_sessionfinish(session,exitstatus):importasyncioasyncio.run(send_dir(dir_path="htmlcov",smtp_hostname="localhost",smtp_port=1025,from_email="yoba@yoba.net",recipient_emails="peka@yoba.net",))ПубликацияДля ручной публикации необходимо задать переменные окруженияPYPI_USERNAMEиPYPI_PASSWORD
Package: aiosendgrid
aiosendgridA simple SendGrid asynchronous client based onhttpx.InstallationpipinstallaiosendgridOr, to include the optional SendGrid helpers support, use:pipinstallaiosendgrid[helpers]UsageWith Mail Helper Classimportaiosendgridfromsendgrid.helpers.mailimportContent,Email,Mail,ToSENDGRID_API_KEY="SG.XXX"from_email=Email("test@example.com")to_email=To("test@example.com")subject="Sending with SendGrid is Fun"content=Content("text/plain","and easy to do anywhere, even with Python")mail=Mail(from_email,to_email,subject,content)asyncwithaiosendgrid.AsyncSendGridClient(api_key=SENDGRID_API_KEY)asclient:response=awaitclient.send_mail_v3(body=mail.get())More info onsendgrid-pythonofficial repository.Without Mail Helper ClassimportaiosendgridSENDGRID_API_KEY="SG.XXX"data={"personalizations":[{"to":[{"email":"test@example.com"}],"subject":"Sending with SendGrid is Fun",}],"from":{"email":"test@example.com"},"content":[{"type":"text/plain","value":"and easy to do anywhere, even with Python"}],}asyncwithaiosendgrid.AsyncSendGridClient(api_key=SENDGRID_API_KEY)asclient:response=awaitclient.send_mail_v3(body=data)
Package: aiosenz
aioSENZ packageAn async Python wrapper for the nVent Raychem SENZ RestAPI.InstallationpipinstallaiosenzOAuth2This package offers anAbstractSENZAuth, where you should handle the OAuth2 tokens and provide a valid access token inget_access_token(). You can useSENZAuthif you don't want to handle the OAuth2 tokens yourself.Grant typeSENZAuthuses the Authorization Code grant type. This requires a Client ID and Client Secret, more information is available inthe RestAPI documentation.ScopesAioSENZ uses therestapiandoffline_accessscope, this is set as default in SENZAuth and should be set in the OAuth2 client if you are using the AbstractSENZAuth class. The OpenID (openid) and OpenID Profile (profile) scopes are not supported, because nVent recommends to use the RestAPI Account instead.ExamplefromasyncioimportrunfromaiosenzimportSENZAuth,SENZAPIimporthttpxasyncdefmain():asyncwithhttpx.AsyncClient()ashttpx_client:senz_auth=SENZAuth(httpx_client,"YOUR_CLIENT_ID","YOUR_CLIENT_SECRET",redirect_uri="http://localhost:8080/auth/callback",)uri,state=awaitsenz_auth.get_authorization_url()print("Authorization URI: ",uri)authorization_response=input("The authorization response URL: ")awaitsenz_auth.set_token_from_authorization_response(authorization_response)senz_api=SENZAPI(senz_auth)thermostats=awaitsenz_api.get_thermostats()forthermostatinthermostats:print(f"{thermostat.name}temperature:{thermostat.current_temperatue}")awaitsenz_auth.close()run(main())
Package: aioserf
AioSerf is an async Python interface to Serf, the decentralised solution
for service discovery and orchestration.It usesanyio <https://github.com/agronholm/anyio>, thus should work withasyncio,trio, andcurio. Hopefully.Installationaioserf requires a running Serf agent. SeeSerf’s agent documentationfor instructions.To install aioserf, run the following command:$pipinstallaioserfor alternatively (you really should be using pip though):$easy_installaioserfor from source:$pythonsetup.pyinstallGetting StartedThese examples require a running async loop.Trio <https://github.com/python-trio/trio>is recommended, thoughasyncioworks too.fromaioserfimportserf_clientasyncwithserf_client()asclient:awaitclient.event('foo','bar')Stream usage:fromaioserfimportserf_clientasyncwithserf_client()asclient:asyncwithclient.stream('*')asstream:asyncforrespinstream:print(resp)Developmentaioserf requires a running Serf agent. SeeSerf’s agent documentationfor instructions.You can run the tests using the following commands:$serfagent--tagfoo=bar&# start serf agent$python3-mpytesttests
Package: aioserver
Installationpip install aioserverGetting StartedStart a test server with the examples in theUsagesection below:make testMake requests against the running test server:curl -v localhost:8080
curl -v localhost:8080/found
curl -v localhost:8080/not-found
curl -v localhost:8080/server-errorUsageCreate ApplicationfromaioserverimportApplicationapp=Application()Basic Routes@app.get('/')asyncdefindex(request):return{'message':'Hello, world!'}@app.get('/found')asyncdeffound(request):return302,{'Location':'https://www.example.com/'},{'message':'Found'}@app.get('/not-found')asyncdefnot_found(request):return404,{'message':'Not Found'}@app.get('/server-error')asyncdefserver_error(request):return500CORS Headers@app.cors('*')@app.get('/cross-origin-resource-sharing')asyncdefcross_origin_resource_sharing(request):return{'message':'Greetings from a different origin!'}@app.cors('*',['X-Custom-Header'])@app.get('/cross-origin-header-sharing')asyncdefcross_origin_header_sharing(request):return200,{'X-Custom-Header':'share-this-header-too'},{'message':'Hello!'}Session Cookiefromaioserver.middlewareimporthours@app.get('/session-cookie')@app.session(max_age=24*hours)asyncdefsession_cookie(request):print(f'session uuid{request.session}')return200,{'message':'Session UUID set as cookie for 24 hours.'}Custom MiddlewareRoute-specific middleware:@app.middlewareasyncdefalways_ok(request,handler):response=awaithandler(request)response.set_status(200,'OK')returnresponse@always_ok@app.get('/not-found-but-still-ok')asyncdefnot_found_but_still_ok(request):return404,{'message':'Not found but still OK!'}Global middleware:asyncdefstrict_transport_security(request,handler):response=awaithandler(request)response.headers['Strict-Transport-Security']='max-age=31536000'returnresponseapp.use(strict_transport_security)Run Applicationapp.run(host='127.0.0.1',port=8080)Changelogv0.2.0Decorator-based request handlersv0.4.0Allow handler to specify HTTP response statusAllow handler to specify additional HTTP headersv0.5.0Serialize XML ElementTree as text/xml responsev0.6.0Decorator-based CORSv0.6.2Fix project descriptionv0.7.0Route-specific middleware using decoratorsGlobal middleware usingapp.use(...)Refactor CORS headers as middlewareAdd session cookie as middlewarev0.7.1Package as a module instead of as a single filev0.7.2Update existing routes when adding new middleware
Package: aioservertiming
aioservertimingTheServer-Timingheader communicates one or more metrics and descriptions for a given request-response cycle. It is used to surface any backend server timing metrics (e.g. database read/write, CPU time, file system access, etc.) in the developer tools in the user's browser or in thePerformanceServerTiminginterface.aioservertimingprovides conventient functions to work with it from aiohttp.InstallationInstallation process as simple as:$ pip install aioservertimingUsageFirst we need to set a middleware to app.fromaiohttpimportwebfromaioservertimingimportserver_timing_mwareapp=web.Applicalion(middlewares=[server_timing_mware])DecoratorContext managerLinksMDN documentationfor Server-TimingThis library onPyPI
Package: aioservice
[![Documentation Status](https://readthedocs.org/projects/aioservice/badge/?version=latest)](http://aioservice.readthedocs.io/en/latest/?badge=latest)
[![Build Status](https://travis-ci.org/denismakogon/aioservice.svg?branch=master)](https://travis-ci.org/denismakogon/aioservice)
[![Coverage](https://codecov.io/gh/denismakogon/aioservice/branch/master/graph/badge.svg)](https://codecov.io/gh/denismakogon/aioservice)AIOServiceAIOService isSet of routines to build web services.AIOService is notA web service, it is a framework that helps to organize web services based on aiohttp.AIOService: HTTPAIOService.HTTP contains set of routines to build HTTP web services with nested versioned API controllers.
It has:Root HTTP serviceVersioned sub-servicesRequest controller handlers routines