updated robots.txt to exclude dirs, updated sitemap