added robots.txt processing in sitemap class