Package: turbocrawler
TurboCrawlerWhat it is?It is a Micro-Framework that you can use to build your crawlers easily, focused in being fast, extremely
customizable, extensible and easy to use, giving you the power to control the crawler behavior.
Provide ways to schedule requests, parse your data asynchronously, extract redirect links from an HTML page.InstallpipinstallturbocrawlerCode ExamplefrompprintimportpprintimportrequestsfromparselimportSelectorfromturbocrawlerimportCrawler,CrawlerRequest,CrawlerResponse,CrawlerRunner,ExecutionInfo,ExtractRuleclassQuotesToScrapeCrawler(Crawler):crawler_name="QuotesToScrape"allowed_domains=['quotes.toscrape']regex_extract_rules=[ExtractRule(r'https://quotes.toscrape.com/page/[0-9]')]time_between_requests=1session:requests.Session@classmethoddefstart_crawler(cls)->None:cls.session=requests.session()@classmethoddefcrawler_first_request(cls)->CrawlerResponse|None:cls.crawler_queue.add(CrawlerRequest(url="https://quotes.toscrape.com/page/9/"))response=cls.session.get(url="https://quotes.toscrape.com/page/1/")returnCrawlerResponse(url=response.url,body=response.text,status_code=response.status_code)@classmethoddefprocess_request(cls,crawler_request:CrawlerRequest)->CrawlerResponse:response=cls.session.get(crawler_request.url)returnCrawlerResponse(url=response.url,body=response.text,status_code=response.status_code)@classmethoddefparse(cls,crawler_request:CrawlerRequest,crawler_response:CrawlerResponse)->None:selector=Selector(crawler_response.body)forquoteinselector.css('div[class="quote"]'):data={"quote":quote.css('span:nth-child(1)::text').get()[1:-1],"author":quote.css('span:nth-child(2)>small::text').get(),"tags_list":quote.css('div[class="tags"]>a::text').getall()}pprint(data)@classmethoddefstop_crawler(cls,execution_info:ExecutionInfo)->None:cls.session.close()CrawlerRunner(crawler=QuotesToScrapeCrawler).run()Understanding the turbocrawler:CrawlerAttributescrawler_namethe name of your crawler, this info will be used byCrawledQueueallowed_domainslist containing all domains that the crawler may add toCrawlerQueueregex_extract_ruleslist containingExtractRuleobjects, the regex passed here will beused to extract all redirect links from an HTML page, EX: 'href="/users"', that you return inCrawlerResponse.bodyIf you let this list empty will not enable the automatic population ofCrawlerQueuefor everyCrawlerResponse.bodytime_between_requestsTime that each request will have to wait before being executedMethodsstart_crawlerShould be used to start a session, webdriver, etc...crawler_first_requestShould be used to make the first request in a site normally the login,
Here could also be used to schedule the first pages to crawl.2 possible Returns:returnCrawlerResponsethe response will be sent toparsemethod and apply follow rule **OBS-1returnNonethe response will not be sent toparsemethodprocess_requestThis method receives all scheduled requests in theCrawlerQueue.addbeing added through manualCrawlerQueue.addor by automatic schedule with regex_extract_rules.Here you must implement all your request logic, cookies, headers, proxy, retries, etc...The method receives aCrawlerRequestand must return aCrawlerResponse.Apply follow rule **OBS-1.process_resposeThis method receives all requests made byprocess_requestHere you can implement any logic, like, scheduling requests,
validating response, remake the requests, etc...
Isn't mandatory to implement this methodparseThis method receives allCrawlerResponsefromcrawler_first_request,process_requestorprocess_resposeHere you can parse your response,
getting the targets fields from HTML and dump the data, in a database for example.stop_crawlerShould be used to close a session, webdriver, etc...OBS:If filledregex_extract_rulesthe redicts specified in the rules will schedule
in theCrawlerQueue, if not filledregex_extract_ruleswill not schedule any request.Order of callsstart_crawlercrawler_first_requestStart loop executing the methods sequentiallyprocess_request->process_response->parse-> loop forever.The loop only stops whenCrawlerQueueis empty.stop_crawlerCrawlerRunnerIs the responsible to run the Crawler, calling the methods in order,
responsible to automatic schedule your requests, and handle the queues.It uses by default:FIFOMemoryCrawlerQueueforCrawlerQueueMemoryCrawledQueueforCrawledQueueBut you can change it using the built-ins queues
inturbocrawler.queuesor creating your own queuesCrawlerQueueCrawlerQueue is where yoursCrawlerRequestare stored
and then will be removed to be processed atprocess_requestCrawledQueueCrawledQueue is where all urls from the processedCrawlerRequestare stored
It prevents to remake a request to the same url, but this behavior can be changed.