deploy robots.txt to prevent crawling until the other issues are sorted out