Automatically trap and block bots that don't obey robots.txt rules