remove the check on robots.txt, it's bad..