Restored max_redirects to default (7). Dropped identifying pdfs by content type
Updated TODO
Big, sloppy commit. URLs to crawl and pdf URLs are now stored in a hash of hashes data structure. If the initial HEAD request for a URL fails, a GET request is attempted in case the server doesn't like HEAD requests. Updated logging. Minor refactoring.