Package: xcrawler
A multi-threaded, open source web crawlerFeaturesUse multiple threads to visit web pagesExtract web page data using XPath expressions or CSS selectorsExtract urls from a web page and visit extracted urlsWrite extracted data to an output fileSet HTTP session parameters such as: cookies, SSL certificates, proxiesSet HTTP request parameters such as: header, body, authenticationDownload files from the urlsSupports Python 2 and Python 3Installationpip install xcrawlerWhen installinglxmllibrary on Windows you may encounterMicrosoft Visual C++ is requirederrors.To installlxmllibrary on Windows:Download and install Microsoft Windows SDK:For Python 2.6, 2.7, 3.0, 3.1, 3.2:Microsoft Windows SDK for .NET Framework 3.5 SP1For Python 3.3, 3.4:Microsoft Windows SDK for .NET Framework 4.0Click the Start Menu, search for and open the command prompt:For Python 2.6, 2.7, 3.0, 3.1, 3.2:CMD ShellFor Python 3.3, 3.4:Windows SDK 7.1 Command PromptInstalllxmlsetenv /x86 /release && SET DISTUTILS_USE_SDK=1 && set STATICBUILD=true && pip install lxmlUsageData and urls are extracted from a web page by a page scraper.To extract data and urls from a web page use the following methods:extractreturns data extracted from a web pagevisitreturns next Pages to be visitedA crawler can be configured before crawling web pages. A user can configure such settings of the crawler as:* the number of threads used to visit web pages* the name of an output file* the request timeoutTo run the crawler call:crawler.run()Examples how to use xcrawler can be found at:https://github.com/cardsurf/xcrawler/tree/master/examplesXPath ExamplefromxcrawlerimportXCrawler,Page,PageScraperclassScraper(PageScraper):defextract(self,page):topics=page.xpath("//a[@class='question-hyperlink']/text()")returntopicsstart_pages=[Page("http://stackoverflow.com/questions/16622802/center-image-within-div",Scraper())]crawler=XCrawler(start_pages)crawler.config.output_file_name="stackoverflow_example_crawler_output.csv"crawler.run()CSS ExamplefromxcrawlerimportXCrawler,Page,PageScraperclassStackOverflowItem:def__init__(self):self.title=Noneself.votes=Noneself.tags=Noneself.url=NoneclassUrlsScraper(PageScraper):defvisit(self,page):hrefs=page.css_attr(".question-summary h3 a","href")urls=page.to_urls(hrefs)return[Page(url,QuestionScraper())forurlinurls]classQuestionScraper(PageScraper):defextract(self,page):item=StackOverflowItem()item.title=page.css_text("h1 a")[0]item.votes=page.css_text(".question .vote-count-post")[0].strip()item.tags=page.css_text(".question .post-tag")[0]item.url=page.urlreturnitemstart_pages=[Page("http://stackoverflow.com/questions?sort=votes",UrlsScraper())]crawler=XCrawler(start_pages)crawler.config.output_file_name="stackoverflow_css_crawler_output.csv"crawler.config.number_of_threads=3crawler.run()File ExamplefromxcrawlerimportXCrawler,Page,PageScraperclassWikimediaItem:def__init__(self):self.name=Noneself.base64=NoneclassEncodedScraper(PageScraper):defextract(self,page):url=page.xpath("//div[@class='fullImageLink']/a/@href")[0]item=WikimediaItem()item.name=url.split("/")[-1]item.base64=page.file(url)returnitemstart_pages=[Page("https://commons.wikimedia.org/wiki/File:Records.svg",EncodedScraper())]crawler=XCrawler(start_pages)crawler.config.output_file_name="wikimedia_file_example_output.csv"crawler.run()Session ExamplefromxcrawlerimportXCrawler,Page,PageScraperfromrequests.authimportHTTPBasicAuthclassScraper(PageScraper):defextract(self,page):returnpage.__str__()start_pages=[Page("http://192.168.1.1/",Scraper())]crawler=XCrawler(start_pages)crawler.config.output_file_name="router_session_example_output.csv"crawler.config.session.headers={"User-Agent":"Custom User Agent","Accept-Language":"fr"}crawler.config.session.auth=HTTPBasicAuth('admin','admin')crawler.run()Request ExamplefromxcrawlerimportXCrawler,Page,PageScraperclassScraper(PageScraper):defextract(self,page):returnpage.__str__()start_page=Page("http://192.168.5.5",Scraper())start_page.request.cookies={"theme":"classic"}crawler=XCrawler([start_page])crawler.config.request_timeout=(5,5)crawler.config.output_file_name="router_request_example_output.csv"crawler.run()DocumentationFor more information about xcrawler see the source code and Python Docstrings:source codeThe documentation can also be accessed at runtime with Python’s built-inhelpfunction:>>>importxcrawler>>>help(xcrawler.Config)# Information about the Config class>>>help(xcrawler.PageScraper.extract)# Information about the extract method of the PageScraper classLicenceGNU GPL v2.0