Reduce load on remote servers

This PR reduces the load on remote servers (those being scanned) by not using the Spider's `use_effective_uris` option, which sends a HEAD request for every link it finds when compiling a list of crawl-able URIs.  Instead, that functionality is put into the downloader class.