Screw robots.txt, I want crawlers!