added robots.txt and sitemap.xml.gz