asses sitemap and modified robots