Now can overwrite default http method creation in
DefaultDocumentFetcher.
Added checks for ignored sitemap/rebots.txt and prevent loading default
if ignored.