Add robots.txt and allow everything to be crawled