Made robots.txt and sitemap_index.xml public

Added urls with the "/robots.txt" or "/sitemap_index.xml" path prefix as
public by default.
Added information about default permissions