Allowed robots to crawl the site