Package: buildaspider
buildaspiderA simple, configurable web crawler written in Python.Designed to be a jumping off point for:understanding and implementing your own crawlerparsing markup withbs4working withrequestsWhile its aims are more educational than industrial, it may still be suitable for crawling sites of moderate size (<1000 unique pages).Written such that it can either be used as-is for small sites, or extended for any number of crawling applications.buildaspideris intended as a platform to learn to build tools for your own quality assurance purposes.InstallationOption 1:pip install buildaspiderOption 2:git clone git@github.com:joedougherty/buildaspider.git
cd buildaspider/
python3 setup.py installExample Config FileA config file isrequired. In addition to the sample given below, you can find an example file inexamples/cgf.ini.[buildaspider]

; login = true
; In order to programatically login, uncomment the line above and ensure login = true
;
; You will also need to ensure that:
;   + the username line is uncommented and set correctly
;   + the password line is uncommented and set correctly
;   + the login_url line is uncommented and set correctly

; username = <USERNAME>
; password = <PASSWORD>
; login_url = http://example.com/login

; Absolute path to directory containing per-run logs
; log_dir = /path/to/logs

; Literal URLs to visit -- there must be at least one!
seed_urls =
    http://httpbin.org/

; List of regex patterns to include
include_patterns =
    httpbin.org

; List of regex patterns to exclude
exclude_patterns =
    ^#$
    ^javascript

max_num_retries = 5Basic UsageOnce the config file is created and ready to go, it is time to create aSpiderinstance.frombuildaspiderimportSpidermyspider=Spider('/path/to/cfg.ini',# These are the default settingsmax_workers=8,time_format="%Y-%m-%d_%H:%M",)myspider.weave()This will start the web crawling process, beginning with the URLs specified inseed_urlsin the config file.LoggingBy default, each run generates four logs:status logbroken links logchecked links logexception links logThe implementation lives in thesetup_loggingmethod of theSpiderbase class:defsetup_logging(self):now=datetime.now().strftime(self.time_format)logging.basicConfig(filename=os.path.join(self.cfg.log_dir,f"spider_{now}.log"),level=logging.INFO,format="%(asctime)s-%(name)s-%(levelname)s-%(message)s",)self.status_logger=logging.getLogger(__name__)self.broken_links_logpath=os.path.join(self.cfg.log_dir,f"broken_links_{now}.log")self.checked_links_logpath=os.path.join(self.cfg.log_dir,f"checked_links_{now}.log")self.exception_links_logpath=os.path.join(self.cfg.log_dir,f"exception_links_{now}.log")There are three rudimentary methods provided that write to each of the above logs:log_checked_linklog_broken_linklog_exception_linkFor example:deflog_checked_link(self,link):append_line_to_log(self.checked_links_logpath,f'{link}')This can be overridden to extend logging capabilities.These methods can also can be overriden to trigger custom behavior when:a link is checkeda broken link is founda link that threw an exception is foundBeyond Basic UsageAdding the Ability to LoginYou can extend the functionality ofbuildaspiderby inheriting from theSpiderclass and overriding methods.This is how you implement the ability for your spider to programmatically login.Here’s the documentation from the baseSpiderclass:deflogin(self):# If your session doesn't require logging in, you can leave this method unimplemented.## Otherwise, this method needs to return an instance of `requests.Session`.## A new session can be obtained by calling `mint_new_session()`.#raiseNotImplementedError("You'll need to implement the login method.")Here’s an example of a fleshed-out login method toPOSTcredentials (as obtained from the config file) to the login_url. (For more details on logging in withrequestssee:https://pybit.es/requests-session.html.)frombuildaspiderimportSpider,mint_new_session,FailedLoginErrorclassMySpider(Spider):deflogin(self):new_session=mint_new_session()login_payload={'username':self.cfg.username,'password':self.cfg.password,}response=new_session.post(self.cfg.login_url,data=login_payload)ifresponse.status_code!=200:raiseFailedLoginError("Login Failed :(")returnresponsemyspider=MySpider('/path/to/cfg.ini')myspider.weave()Providing Custom Functionality by Attaching to Event HooksThere are a few events that occur during the crawling process that you may want to attach some additional functionality to.There are pre-visit and post-visit methods you can override/extend.EventMethodlink visit is about to begin.pre_visit_hook()link visit is about to end.post_visit_hook()a link has been marked as checked.log_checked_link()a link has been marked as broken.log_broken_link()a link has been marked as causing an exception.log_exception_link()crawling is complete.cleanup()Spider.pre_visit_hook()provides the ability to run code when.visit()is called. Code specified in.pre_visit_hook()will execute prior to library-provided functionality in.visit().Spider.post_visit_hook()provides the ability to run code right before.visit()finishes.The overridden methods.pre_visit_hook()and.post_visit_hook()ought to pass inlinkin order to keep the current link in scope and available as a variable with that name.You may choose to store visited links in some custom container:custom_visited_links=list()defpre_visit_hook(self,link):# The `link` being referenced here# is the link about to be visitedcustom_visited_links.append(link)NOTE:this provides direct access to the currentLinkobject in scope.A safe strategy is to make a copy of the currentLinkusingdeepcopy.fromcopyimportdeepcopycustom_visited_links=list()defpre_visit_hook(self,link):current_link_copy=deepcopy(link)custom_visited_links.append(current_link_copy)Extending/Overriding Pre-Defined EventsBy default, broken links are logged to the location specified byself.broken_links_logpath.We can see this in theSpiderclass:deflog_broken_link(self,link):append_line_to_log(self.broken_links_logpath,f'{link}::{link.http_code}')What if you want toextend(not merely override) the functionality of.log_broken_link()?deflog_broken_link(self,link):super().log_broken_link(link)# You've now retained the original functionality# by running the method as defined on the parent instance# Perhaps now you want to:#   + cache this value?#   + run some action(s) as a result of this event firing?#   + ???Running the Test SuiteNOTE:You will need to ensure that thelog_dirconfig file field is set correctly before you run the test suite.cd tests/
pytestAdditional ResourcesOfficial Retry Documentationhttps://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#module-urllib3.util.retryAdvanced usage of Python requests - timeouts, retries, hookshttps://findwork.dev/blog/advanced-usage-python-requests-timeouts-retries-hooks/#retry-on-failurePython stdlib Logging: basicConfighttps://docs.python.org/3.8/library/logging.html#logging.basicConfigBFS / FIFO Queuehttps://en.wikipedia.org/wiki/Breadth-first_search#PseudocodePython: A quick introduction to the concurrent.futures modulehttp://masnun.com/2016/03/29/python-a-quick-introduction-to-the-concurrent-futures-module.htmlUsing Python Requests on a Page Behind a Loginhttps://pybit.es/requests-session.htmlThe Offical collections.deque Documentationhttps://docs.python.org/3.8/library/collections.html#collections.deque