Add robots.txt, improve sitemap and atom.