UNLSiteMaster/site_master — pull request #58 by mfairchild365 (2014-03-17 20:41:59+00:00)
Title: Reduce load on remote servers

This PR reduces the load on remote servers (those being scanned) by not using the Spider's `use_effective_uris` option, which sends a HEAD request for every link it finds when compiling a list of crawl-able URIs. Instead, that functionality is put into the downloader class.