Package: nyawc
Not Your Average Web CrawlerN.Y.A.W.C is a Python library that enables you to test your payload against all requests of a certain domain. It crawls all requests (e.g. GET, POST or PUT) in the specified scope and keeps track of the request and response data. During the crawling process the callbacks enable you to insert your payload at specific places and test if they worked.Table of contentsInstallationCrawling flowDocumentationMinimal implementationTestingIssuesLicenseInstallationFirst make sure you’re onPython 2.7/3.3or higher. Then run the command below to install N.Y.A.W.C.$ pip install--upgradenyawcCrawling flowYou can define your startpoint (a request) and the crawling scope and then start the crawler.The crawler repeatedly starts the first request in the queue untilmax threadsis reached.The crawler adds all requests found in the response to the end of the queue (except duplicates).The crawler goes back to step #2 to spawn new requests repeatedly untilmax threadsis reached.Please note that if the queue is empty and all crawler threads are finished, the crawler will stop.DocumentationPlease refer to thedocumentationor theAPIfor all the information about N.Y.A.W.C.Minimal implementationYou can use the callbacks inexample_minimal.pyto run your own exploit against the requests. If you want an example of automated exploit scanning, please take a look atACSTIS(it uses N.Y.A.W.C to scan for AngularJS client-side template injection vulnerabilities).You can also use thekitchen sink(which contains all the functionalities from N.Y.A.W.C.) instead of the example below. The code below is a minimal implementation of N.Y.A.W.C.$ python example_minimal.py$ python-uexample_minimal.py > output.log# example_minimal.pyfromnyawc.OptionsimportOptionsfromnyawc.QueueItemimportQueueItemfromnyawc.CrawlerimportCrawlerfromnyawc.CrawlerActionsimportCrawlerActionsfromnyawc.http.RequestimportRequestdefcb_crawler_before_start():print("Crawler started.")defcb_crawler_after_finish(queue):print("Crawler finished.")print("Found "+str(len(queue.get_all(QueueItem.STATUS_FINISHED)))+" requests.")defcb_request_before_start(queue,queue_item):print("Starting:{}".format(queue_item.request.url))returnCrawlerActions.DO_CONTINUE_CRAWLINGdefcb_request_after_finish(queue,queue_item,new_queue_items):print("Finished:{}".format(queue_item.request.url))returnCrawlerActions.DO_CONTINUE_CRAWLINGoptions=Options()options.callbacks.crawler_before_start=cb_crawler_before_start# Called before the crawler starts crawling. Default is a null route.options.callbacks.crawler_after_finish=cb_crawler_after_finish# Called after the crawler finished crawling. Default is a null route.options.callbacks.request_before_start=cb_request_before_start# Called before the crawler starts a new request. Default is a null route.options.callbacks.request_after_finish=cb_request_after_finish# Called after the crawler finishes a request. Default is a null route.crawler=Crawler(options)crawler.start_with(Request("https://finnwea.com/"))TestingThe testing can and will automatically be done byTravis CIon every push to the master branch. If you want to manually run the unit tests, use the command below.$ python-munittest discoverIssuesIssues or new features can be reported via the GitHub issue tracker. Please make sure your issue or feature has not yet been reported by anyone else before submitting a new one.LicenseNot Your Average Web Crawler (N.Y.A.W.C) is open-sourced software licensed under theMIT license.