Package: acrawler
🔍 A powerful web-crawling framework, based on aiohttp.FeatureWrite your crawler in one Python script with asyncioSchedule task with priority, fingerprint, exetime, recrawl…Middleware: add handlers before or after task’s executionSimple shortcuts to speed up scriptingParse html conveniently withParselParse with rules and chained processorsSupport JavaScript/browser-automation withpyppeteerStop and Resume: crawl periodically and persistentlyDistributed work support with RedisInstallationTo install, simply use pip:$pipinstallacrawler(Optional)$pipinstalluvloop#(only Linux/macOS, for faster asyncio event loop)$pipinstallaioredis#(if you need Redis support)$pipinstallmotor#(if you need MongoDB support)$pipinstallaiofiles#(if you need FileRequest)DocumentationDocumentation and tutorial are available online athttps://acrawler.readthedocs.io/and in thedocsdirectory.Sample CodeScrape imdb.comfromacrawlerimportCrawler,Request,ParselItem,Handler,register,get_loggerclassMovieItem(ParselItem):log=Truecss={# just some normal css rules# see Parsel for detailed information"date":".subtext a[href*=releaseinfo]::text","time":".subtext time::text","rating":"span[itemprop=ratingValue]::text","rating_count":"span[itemprop=ratingCount]::text","metascore":".metacriticScore span::text",# if you provide a list with additional functions,# they are considered as field processor function"title":["h1::text",str.strip],# the following four fules is for getting all matching values# the rule starts with [ and ends with ] comparing to normal rules"genres":"[.subtext a[href*=genres]::text]","director":"[h4:contains(Director) ~ a[href*=name]::text]","writers":"[h4:contains(Writer) ~ a[href*=name]::text]","stars":"[h4:contains(Star) ~ a[href*=name]::text]",}classIMDBCrawler(Crawler):config={"MAX_REQUESTS":4,"DOWNLOAD_DELAY":1}asyncdefstart_requests(self):yieldRequest("https://www.imdb.com/chart/moviemeter",callback=self.parse)defparse(self,response):yield fromresponse.follow(".lister-list tr .titleColumn a::attr(href)",callback=self.parse_movie)defparse_movie(self,response):url=response.url_stryieldMovieItem(response.sel,extra={"url":url.split("?")[0]})@register()classHorrorHandler(Handler):family="MovieItem"logger=get_logger("horrorlog")asyncdefhandle_after(self,item):ifitem["genres"]and"Horror"initem["genres"]:self.logger.warning(f"({item['title']}) is a horror movie!!!!")@MovieItem.bind()defprocess_time(value):# a self-defined field processing function# process time to minutes# '3h 1min' -> 181ifvalue:res=0segs=value.split(" ")forseginsegs:ifseg.endswith("min"):res+=int(seg.replace("min",""))elifseg.endswith("h"):res+=60*int(seg.replace("h",""))returnresreturnvalueif__name__=="__main__":IMDBCrawler().run()Scrape quotes.toscrape.com# Scrape quotes from http://quotes.toscrape.com/fromacrawlerimportParser,Crawler,ParselItem,Requestlogger=get_logger("quotes")classQuoteItem(ParselItem):log=Truedefault={"type":"quote"}css={"author":"small.author::text"}xpath={"text":['.//span[@class="text"]/text()',lambdas:s.strip("“")[:20]]}classAuthorItem(ParselItem):log=Truedefault={"type":"author"}css={"name":"h3.author-title::text","born":"span.author-born-date::text"}classQuoteCrawler(Crawler):main_page=r"quotes.toscrape.com/page/\d+"author_page=r"quotes.toscrape.com/author/.*"parsers=[Parser(in_pattern=main_page,follow_patterns=[main_page,author_page],item_type=QuoteItem,css_divider=".quote",),Parser(in_pattern=author_page,item_type=AuthorItem),]asyncdefstart_requests(self):yieldRequest(url="http://quotes.toscrape.com/page/1/")if__name__=="__main__":QuoteCrawler().run()Seeexamples.TodoReplace parsel with parselxclean redundant handlersCralwer’s name for distinguishingUse dynaconf as configuration managerAdd delta_key support for requestMonitor all crawlers in webWrite detailed DocumentationTesting