Adding robots.txt and capturing bad bots