Making the web crawler, sitemap-aware. gzipped versions of sitemaps handled as well!