Package: python-testing-crawler
Python Testing Crawler   :snake: :stethoscope: :spider:A crawler for automated functional testing of a web applicationCrawling a server-side-rendered web application is alow costway to getlow qualitytest coverage of your JavaScript-light web application.If you have only partial test coverage of your routes, but still want to protect against silly mistakes, then this is for you.Features:Selectively spider pages and resources, or just request themSubmit forms, and control what values to sendIgnore links by source using CSS selectorsFail fast or collect many errorsConfigurable using straightforward rulesWorks with the test clients forFlask(incFlask-WebTest),DjangoandWebTest.Why should I use this?Here's an example:Flaskr, the Flask tutorial application has166 lines of test codeto achieve 100% test coverage.Using Python Testing Crawlerin a similar way to the Usage example below, we can hit 73% with very little effort. Disclaimer: Of course! It's not the same quality or utility of testing! But it is better than no tests, a complement to hand-written unit or functional tests and a useful stopgap.Installation$ pip install python-testing-crawlerUsageCreate a crawler using your framework's existing test client, tell it where to start and what rules to obey, then set it off:frompython_testing_crawlerimportCrawlerfrompython_testing_crawlerimportRule,Request,Ignore,Allowdeftest_crawl_all():client=## ... existing testing client## ... any setup ...crawler=Crawler(client=my_testing_client,initial_paths=['/'],rules=[Rule("a",'/.*',"GET",Request()),])crawler.crawl()This will crawl all anchor links to relative addresses beginning "/". Any exceptions encountered will be collected and presented at the end of the crawl. Formore powersee the Rules section below.If you need to authorise the client's session, e.g. login, then you should that before creating the Crawler.It is also a good idea to create enough data, via fixtures or otherwise, to expose enough endpoints.How do I setup a test client?It depends on your framework:Flask:https://flask.palletsprojects.com/en/1.1.x/testing/Django:https://docs.djangoproject.com/en/3.0/topics/testing/tools/Crawler OptionsParamDescriptioninitial_pathslist of paths/URLs to start fromruleslist of Rules to control the crawler; see belowpath_attrslist of attribute names to extract paths/URLs from; defaults to "href" -- include "src" if you want to check e.g.<link>,<script>or even<img>ignore_css_selectorsany elements matching this list of CSS selectors will be ignored when extracting linksignore_form_fieldslist of form input names to ignore when determining the identity/uniqueness of a form. Include CSRF token field names here.max_requestsCrawler will raise an exception if this limit is exceededcapture_exceptionsupon encountering an exception, keep going and fail at the end of the crawl instead of during (defaultTrue)output_summaryprint summary statistics and any captured exceptions and tracebacks at the end of the crawl (defaultTrue)should_process_handlerslist of "should process" handlers; see Handlers sectioncheck_response_handlerslist of "check response" handlers; see Handlers sectionRulesThe crawler has to be told what URLs to follow, what forms to post and what to ignore, using Rules.Rules are made of four parameters:Rule(<source element regex>, <target URL/path regex>, <HTTP method>, <action to take>)These are matched against every HTML element that the crawler encounters, with the last matching rule winning.Actions must be one of the following objects:Request(only=False, params=None)-- follow a link or submit a formonly=Truewill retrieve a page/resource butnotspider its links.the dictparamsallows you to specifyoverridesfor a form's default valuesIgnore()-- do nothing / skipAllow(status_codes)-- allow a HTTP status in the supplied list, i.e. do not consider it an error.Example RulesFollow all local/relative linksHYPERLINKS_ONLY_RULE_SET=[Rule('a','/.*','GET',Request()),Rule('area','/.*','GET',Request()),]Request but do not spider all linksREQUEST_ONLY_EXTERNAL_RULE_SET=[Rule('a','.*','GET',Request(only=True)),Rule('area','.*','GET',Request(only=True)),]This is useful for finding broken links.  You can also check<link>tags from the<head>if you include the following ruleplusset the Crawler'spath_attrsto("HREF", "SRC").Rule('link', '.*', 'GET', Request())Submit forms with GET or POSTSUBMIT_GET_FORMS_RULE_SET=[Rule('form','.*','GET',Request())]SUBMIT_POST_FORMS_RULE_SET=[Rule('form','.*','POST',Request())]Forms are submitted with their default values, unless overridden usingRequest(params={...})for a specific form target or excluded using (globally) using theignore_form_fieldsparameter toCrawler(necessary for e.g. CSRF token fields).Allow some routes to failPERMISSIVE_RULE_SET=[Rule('.*','.*','GET',Allow([*range(400,600)])),Rule('.*','.*','POST',Allow([*range(400,600)]))]If any HTTP error (400-599) is encountered for any request, allow it; do not error.Crawl GraphThe crawler builds up a graph of your web application. It can be interrogated viacrawler.graphwhen the crawl is finished.Seethe graph modulefor the defintion ofNodeobjects.HandlersTwo hooks points are provided. These operate onNodeobjects (see above).Whether to process a NodeUsingshould_process_handlers, you can register functions that take aNodeand return aboolof whether the Crawler should "process" -- follow a link or submit a form -- or not.Whether a response is acceptableUsingcheck_response_handlers, you can register functions that take aNodeand response object (specific to your test client) and return a bool of whether the response should constitute an error.If your function returnsTrue, the Crawler with throw an exception.ExamplesThere are currently Flask and Django examples inthe tests.Seehttps://github.com/python-testing-crawler/flaskrfor an example of integrating into an existing application, using Flaskr, the Flask tutorial application.