Package: pyscalpel
PyscalpelYour easy-to-use, fast and powerful web scraping library.Why?I already knewscrapywhich is the reference in python for web scraping. But
two things bothered me.I feel like scrapy cannot integrate into an existing project, you need to treat your web scraping stuff like a project
on its own.Usage ofTwistedwho is a veteran in asynchronous programming, but I think
that there are better asynchronous frameworks today. Note that this second point is not true anymore as I'm writing
the document since scrapy adds support forasyncioAfter having made this observation I decided to create pyscalpel. And let's be honest, I also want to have my own web
scraping library, and it is fun to write one ;)Installationpipinstallpyscalpel# to only use the asyncio backendpipinstallpyscalpel[gevent]# to install the gevent backendpipinstallpyscalpel[trio]# to installl the trio backendpipinstallpyscalpel[full]# to install all the backendsIf you know aboutpoetryyou can use it instead of pip.poetryaddpyscalpel# to only use the asyncio backendpoetryaddpyscalpel[gevent]# to install the gevent backendpoetryaddpyscalpel[trio]# to install the trio backendpoetryaddpyscalpel[full]# to install all the backendspyscalpel works starting frompython 3.7, it relies on robust packages:configuror: A configuration toolkit.httpx: A modern http client.selenium: A library for controlling a browser.gevent: An asynchronous framework using the synchronous way. (optional)trio: A modern asynchronous framework usingasync/awaitsyntax. (optional)anyio: An asynchronous networking and concurrency library that works on top of
either asyncio or trio.parsel: A library elements in HTML/XML documents.attrs: A library helping to write classes without pain.fake-useragent: A simple library to fake a user agent.rfc3986: A library for url parsing and validation.msgpack: A library allowing for fast serialization/deserialization of data
structures.DocumentationThe documentation is available athttps://scalpel.readthedocs.io/en/latest/.UsageTo give you an overview of what can be done, this is a simple example of quote scraping. Don't hesitate to look at the
examples folder for more snippets to look at.with geventfrompathlibimportPathfromscalpelimportConfigurationfromscalpel.greenimportStaticSpider,StaticResponse,read_mpdefparse(spider:StaticSpider,response:StaticResponse)->None:forquoteinresponse.xpath('//div[@class="quote"]'):data={'message':quote.xpath('./span[@class="text"]/text()').get(),'author':quote.xpath('./span/small/text()').get(),'tags':quote.xpath('./div/a/text()').getall()}spider.save_item(data)next_link=response.xpath('//nav/ul/li[@class="next"]/a').xpath('@href').get()ifnext_linkisnotNone:response.follow(next_link)if__name__=='__main__':backup=Path(__file__).parent/'backup.mp'config=Configuration(backup_filename=f'{backup}')spider=StaticSpider(urls=['http://quotes.toscrape.com'],parse=parse,config=config)spider.run()print(spider.statistics())# you can do whatever you want with the resultsforquote_datainread_mp(filename=backup,decoder=spider.config.msgpack_decoder):print(quote_data)with anyiofrompathlibimportPathimportanyiofromscalpelimportConfigurationfromscalpel.any_ioimportStaticResponse,StaticSpider,read_mpasyncdefparse(spider:StaticSpider,response:StaticResponse)->None:forquoteinresponse.xpath('//div[@class="quote"]'):data={'message':quote.xpath('./span[@class="text"]/text()').get(),'author':quote.xpath('./span/small/text()').get(),'tags':quote.xpath('./div/a/text()').getall()}awaitspider.save_item(data)next_link=response.xpath('//nav/ul/li[@class="next"]/a').xpath('@href').get()ifnext_linkisnotNone:awaitresponse.follow(next_link)asyncdefmain():backup=Path(__file__).parent/'backup.mp'config=Configuration(backup_filename=f'{backup}')spider=StaticSpider(urls=['http://quotes.toscrape.com'],parse=parse,config=config)awaitspider.run()print(spider.statistics())# you can do whatever you want with the resultsasyncforiteminread_mp(backup,decoder=spider.config.msgpack_decoder):print(item)if__name__=='__main__':# by default, this will run the asyncio backend, if you want the trio backend, you must first install the trio# package and replace the following line with: anyio.run(main, backend='trio').anyio.run(main)Known limitationspyscalpel aims to handle SPA (single page application) through the use of selenium. However, due to the synchronous nature
of selenium, it is hard to leverage anyio and gevent asynchronous feature. You will notice that theselenium spideris
slower than thestatic spider. For more information look at the documentation.Warningpyscalpel is a young project, so it is expected to have breaking changes in the api without respecting thesemverprinciple. It is recommended to pin the version you are using for now.